Sep 15 16:24:26.212262 ip-10-0-130-221 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 16:24:26.212273 ip-10-0-130-221 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 16:24:26.212282 ip-10-0-130-221 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 16:24:26.212482 ip-10-0-130-221 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 16:24:36.303392 ip-10-0-130-221 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 16:24:36.303413 ip-10-0-130-221 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ae186f4b90314441aac6bef11597e70a --
Sep 15 16:26:34.281221 ip-10-0-130-221 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 16:26:34.676552 ip-10-0-130-221 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:34.676552 ip-10-0-130-221 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 16:26:34.676552 ip-10-0-130-221 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:34.676552 ip-10-0-130-221 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 16:26:34.676552 ip-10-0-130-221 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:34.681187 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.680993    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 16:26:34.685068 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685046    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:34.685068 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685064    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:34.685068 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685070    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:34.685068 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685074    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685078    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685082    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685086    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685091    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685094    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685101    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685107    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685112    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685116    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685120    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685124    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685128    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685132    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685136    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685140    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685144    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685148    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685152    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685156    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:34.685306 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685160    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685163    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685167    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685171    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685175    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685179    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685183    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685188    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685192    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685200    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685205    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685209    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685214    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685218    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685223    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685227    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685231    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685236    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685240    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685243    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685248    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:34.686160 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685252    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685256    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685260    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685264    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685268    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685274    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685279    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685283    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685287    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685291    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685296    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685300    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685303    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685307    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685311    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685315    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685319    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685323    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685327    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:34.687079 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685331    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685336    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685342    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685346    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685351    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685355    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685360    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685365    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685369    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685374    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685377    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685382    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685386    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685390    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685394    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685400    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685404    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685409    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685413    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:34.687904 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685417    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685421    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685426    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685430    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685434    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685440    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685445    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685449    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685454    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685458    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685462    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685467    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685473    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685478    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685483    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685488    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.685492    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686676    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686685    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:34.688850 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686691    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686696    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686700    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686705    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686709    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686717    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686724    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686730    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686735    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686739    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686744    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686748    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686754    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686758    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686763    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686767    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686771    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686775    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686779    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686783    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:34.689691 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686787    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686791    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686796    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686800    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686804    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686808    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686812    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686816    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686821    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686825    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686829    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686833    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686838    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686843    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686847    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686851    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686855    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686859    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686863    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686867    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:34.690290 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686872    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686877    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686881    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686886    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686890    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686895    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686900    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686904    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686908    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686915    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686920    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686924    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686949    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686953    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686956    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686960    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686964    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686968    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686972    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686976    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:34.690822 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686981    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686988    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686994    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.686998    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687003    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687007    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687012    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687016    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687021    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687025    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687029    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687033    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687038    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687042    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687046    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687050    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687055    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687059    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687064    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:34.691400 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687068    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687074    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687078    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687081    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687085    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687089    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687092    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687097    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687101    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687105    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687109    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687113    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687118    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687122    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687126    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687130    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687135    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687140    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687760    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687768    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:34.691889 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687772    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687777    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687781    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687785    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687790    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687793    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687798    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687802    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687806    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687810    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687814    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687819    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687829    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687834    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687838    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687843    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687850    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687856    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687860    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:34.692617 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687865    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687869    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687876    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687881    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687886    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687890    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687894    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687898    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687902    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687906    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687913    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687917    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687921    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687925    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687951    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687956    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687960    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687964    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687968    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687972    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:34.693119 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687976    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687980    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687985    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687989    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.687993    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688003    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688008    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688012    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688017    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688021    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688025    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688029    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688033    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688037    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688041    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688045    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688049    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688053    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688057    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688062    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:34.693694 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688066    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688070    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688076    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688080    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688084    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688088    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688092    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688096    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688101    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688105    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688113    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688117    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688121    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688125    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688131    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688136    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688140    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688146    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688152    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688156    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:34.694240 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688160    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688164    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688168    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688172    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688177    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688181    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688185    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688189    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688193    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688197    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688201    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688205    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688209    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688213    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688219    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688223    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688227    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.688232    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688335    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688377    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 16:26:34.694742 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688387    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688394    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688400    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688405    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688410    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688416    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688421    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688425    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688430    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688437    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688442    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688448    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688452    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688457    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688461    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688466    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688472    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688476    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688481    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688486    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688491    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688497    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688502    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 16:26:34.695251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688507    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688512    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688517    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688523    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688528    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688533    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688538    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688545    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688550    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688555    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688560    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688564    2564 flags.go:64] FLAG: --enable-server="true"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688569    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688578    2564 flags.go:64] FLAG: --event-burst="100"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688583    2564 flags.go:64] FLAG: --event-qps="50"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688588    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688593    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688597    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688604    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688609    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688614    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688620    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688624    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688629    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688634    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 16:26:34.696417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688639    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688643    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688648    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688652    2564 flags.go:64] FLAG: --feature-gates=""
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688658    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688663    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688668    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688673    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688678    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688683    2564 flags.go:64] FLAG: --help="false"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688689    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-130-221.ec2.internal"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688694    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688699    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688703    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688709    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688715    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688719    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688724    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688728    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688733    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688738    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688743    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688748    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688753    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 16:26:34.697104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688757    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688764    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688769    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688773    2564 flags.go:64] FLAG: --lock-file=""
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688777    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688783    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688788    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688796    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688801    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688806    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688810    2564 flags.go:64] FLAG: --logging-format="text"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688814    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688820    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688825    2564 flags.go:64] FLAG: --manifest-url=""
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688829    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688836    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688840    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688846    2564 flags.go:64] FLAG: --max-pods="110"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688852    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688857    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688869    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688874    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688879    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688883    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688887    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 16:26:34.697712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688898    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688903    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688907    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688912    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688917    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688925    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688947    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688952    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688957    2564 flags.go:64] FLAG: --port="10250"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688962    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688966    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04722e1bf4605726b"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688971    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688976    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688982    2564 flags.go:64] FLAG: --register-node="true"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688987    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688993    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.688997    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689002    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689006    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689012    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689017    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689022    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689026    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689031    2564 flags.go:64] FLAG: --runonce="false"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689035    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 16:26:34.698381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689040    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689045    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689049    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689054    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689058    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689063    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689068    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689072    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689077    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689081    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689086    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689090    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689095    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689099    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689109    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689113    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689118    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689124    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689129    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689144    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689149    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689155    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689159    2564 flags.go:64] FLAG: --v="2"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689166    2564 flags.go:64] FLAG: --version="false"
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689172    2564 flags.go:64] FLAG: --vmodule=""
Sep 15 16:26:34.699001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689178    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.689183    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689375    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689382    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689387    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689391    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689395    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689400    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689404    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689409    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689413    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689417    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689421    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689425    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689430    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689434    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689438    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689443    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689447    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689451    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:34.699605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689455    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689460    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689464    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689469    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689474    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689478    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689482    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689486    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689490    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689503    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689508    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689513    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689519    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689525    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689530    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689535    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689541    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689545    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689549    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:34.700177 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689553    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689557    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689561    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689565    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689569    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689573    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689577    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689581    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689585    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689590    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689594    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689598    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689602    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689606    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689609    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689614    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689618    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689622    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689626    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689630    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:34.700645 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689634    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689638    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689642    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689647    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689651    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689655    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689659    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689663    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689667    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689671    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689677    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689683    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689687    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689692    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689696    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689700    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689703    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689707    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689711    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:34.701157 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689715    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689719    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689723    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689727    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689732    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689736    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689740    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689744    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689752    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689758    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689764    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689768    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689772    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689777    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689781    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689785    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689789    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689801    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689806    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689810    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:34.701622 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689814    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689818    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.689822    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.690727    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.698001    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.698017    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.698747    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.698759    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698814    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698819    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698824    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698828    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698831    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698834    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:34.702169 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698837    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698840    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698842    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698845    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698847    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698850    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698852    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698855    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698858    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698860    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698863    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698866    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698868    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698871    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698873    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698876    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698878    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698881    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698883    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698886    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:34.702532 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698888    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698891    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698894    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698897    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698900    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698903    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698906    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698909    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698912    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698914    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698917    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698919    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698922    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698924    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698927    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698946    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698949    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698951    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698954    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698956    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:34.703039 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698959    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698963    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698967    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698970    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698972    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698975    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698977    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698979    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698982    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698984    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698987    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698990    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698992    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698995    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.698998    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699001    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699005    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699008    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699010    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:34.703526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699013    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699016    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699018    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699020    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699023    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699026    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699028    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699030    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699033    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699035    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699038    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699040    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699043    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699045    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699048    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699050    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699053    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699055    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699058    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699060    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:34.704056 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699063    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699066    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699068    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699072    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699075    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699078    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699081    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699085    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699088    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699091    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699094    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699096    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699099    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699101    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:34.704549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.699106    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699223    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699228    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699231    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699234    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699236    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699239    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699242    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699244    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699247    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699250    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699252    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699255    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699257    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699260    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699262    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699265    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699267    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699270    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699272    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:34.704905 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699275    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699277    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699280    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699283    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699285    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699288    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699290    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699293    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699295    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699298    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699302    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699306    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699308    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699311    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699314    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699317    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699320    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699322    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699325    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:34.705417 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699328    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699330    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699333    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699336    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699338    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699340    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699343    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699345    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699348    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699350    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699352    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699355    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699357    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699360    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699363    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699366    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699369    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699371    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699373    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:34.705881 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699376    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699379    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699381    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699384    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699387    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699390    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699392    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699395    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699397    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699400    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699402    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699404    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699407    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699409    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699412    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699414    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699417    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699420    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699422    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:34.706364 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699424    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699427    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699429    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699432    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699434    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699437    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699439    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699443    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699446    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699449    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699452    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699455    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699458    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699460    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699463    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699466    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699468    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699471    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699474    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:34.706899 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699477    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699479    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699482    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:34.699484    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.699489    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.699600    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.701712    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.702731    2564 server.go:1019] "Starting client certificate rotation"
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.702837    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.703600    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 16:26:34.707649 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.705437    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 16:26:34.709712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.709685    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 16:26:34.722007 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.721959    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 15 16:26:34.725013 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.724998    2564 log.go:25] "Validated CRI v1 image API"
Sep 15 16:26:34.725955 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.725921    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 16:26:34.729168 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.729149    2564 fs.go:135] Filesystem UUIDs: map[0aa8bd3d-539e-482e-b8e9-72f2eeb63c45:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9a0f1cb3-0f91-4b53-a229-742703206771:/dev/nvme0n1p3]
Sep 15 16:26:34.729228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.729168    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 16:26:34.730393 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.730375    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 16:26:34.734225 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.734037    2564 manager.go:216] Machine: {Timestamp:2026-09-15 16:26:34.732759218 +0000 UTC m=+0.335176829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091372 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec223f0e78fa8ce2eae9c08fcf1a6625 SystemUUID:ec223f0e-78fa-8ce2-eae9-c08fcf1a6625 BootID:ae186f4b-9031-4441-aac6-bef11597e70a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:08:19:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:08:19:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:1d:86:a4:98:6f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 16:26:34.734225 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.734224    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 16:26:34.734333 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.734297    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 16:26:34.737304 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.737283    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 16:26:34.737442 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.737306    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-221.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 16:26:34.737998 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.737989    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 16:26:34.738036 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.738000    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 16:26:34.738036 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.738008    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 16:26:34.738565 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.738556    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 16:26:34.739274 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.739265    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 16:26:34.743024 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743013    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 16:26:34.743136 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743126    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 16:26:34.743181 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743155    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 16:26:34.743181 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743173    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 16:26:34.743259 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743187    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 16:26:34.743259 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743196    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 15 16:26:34.743259 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743206    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 16:26:34.743560 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743549    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 16:26:34.743601 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.743568    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 16:26:34.745656 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.745636    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 16:26:34.749280 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.749260    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 16:26:34.749356 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.749292    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 16:26:34.750975 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.750955    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 16:26:34.751068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.750983    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 16:26:34.751068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.750995    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 16:26:34.751068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751004    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 16:26:34.751068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751021    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 16:26:34.751068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751030    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 16:26:34.751068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751053    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 16:26:34.751068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751064    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 16:26:34.751320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751079    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 16:26:34.751320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751092    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 16:26:34.751320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751131    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 16:26:34.751320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751141    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 16:26:34.751857 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751846    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 16:26:34.751902 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.751859    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 16:26:34.755753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.755737    2564 server.go:1268] "Started kubelet"
Sep 15 16:26:34.755837 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.755805    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 16:26:34.755881 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.755821    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 16:26:34.755922 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.755891    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 16:26:34.756301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.756279    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-knls6"
Sep 15 16:26:34.756515 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.756501    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-221.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 16:26:34.756516 ip-10-0-130-221 systemd[1]: Started Kubernetes Kubelet.
Sep 15 16:26:34.756873 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.756824    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 16:26:34.756970 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.756848    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 16:26:34.757271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.757161    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 16:26:34.757955 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.757926    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 16:26:34.761453 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.761426    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 16:26:34.761613 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.761598    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 16:26:34.761668 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.761636    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-knls6"
Sep 15 16:26:34.761668 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.760732    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-221.ec2.internal.18d58b4e28ac05dd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-221.ec2.internal,UID:ip-10-0-130-221.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-221.ec2.internal,},FirstTimestamp:2026-09-15 16:26:34.755712477 +0000 UTC m=+0.358130087,LastTimestamp:2026-09-15 16:26:34.755712477 +0000 UTC m=+0.358130087,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-221.ec2.internal,}"
Sep 15 16:26:34.762073 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.762056    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 16:26:34.762199 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.762182    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:34.762632 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.762615    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 16:26:34.762632 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.762624    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 16:26:34.762757 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.762643    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 16:26:34.762823 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.762809    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 16:26:34.762823 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.762823    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 16:26:34.763045 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.763031    2564 factory.go:55] Registering systemd factory
Sep 15 16:26:34.763090 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.763048    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 15 16:26:34.763258 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.763248    2564 factory.go:153] Registering CRI-O factory
Sep 15 16:26:34.763258 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.763258    2564 factory.go:223] Registration of the crio container factory successfully
Sep 15 16:26:34.763323 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.763296    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 16:26:34.763323 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.763309    2564 factory.go:103] Registering Raw factory
Sep 15 16:26:34.763382 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.763360    2564 manager.go:1196] Started watching for new ooms in manager
Sep 15 16:26:34.763664 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.763652    2564 manager.go:349] Starting recovery of all containers
Sep 15 16:26:34.767695 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.767669    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-221.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 16:26:34.767825 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.767802    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 16:26:34.774531 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.774499    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 16:26:34.775231 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.775214    2564 manager.go:354] Recovery completed
Sep 15 16:26:34.779577 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.779564    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:34.783247 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.783226    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:34.783310 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.783263    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:34.783310 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.783274    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:34.784785 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.784766    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 16:26:34.784785 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.784786    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 16:26:34.784896 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.784800    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:34.784896 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.784844    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 16:26:34.786212 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.786192    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:34.786257 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.786229    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:34.786257 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.786241    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:34.786897 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.786886    2564 policy_none.go:49] "None policy: Start"
Sep 15 16:26:34.786949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.786902    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 16:26:34.786949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.786915    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 16:26:34.788779 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.788768    2564 policy_none.go:47] "Start"
Sep 15 16:26:34.826546 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.826531    2564 manager.go:337] "Starting Device Plugin manager"
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.826571    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.826585    2564 server.go:87] "Starting device plugin registration server"
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.826903    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.826915    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.826995    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.827067    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.827077    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.827583    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 16:26:34.847330 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.827627    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:34.892119 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.892094    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 16:26:34.892250 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.892153    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 16:26:34.892250 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.892180    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 16:26:34.892315 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.892228    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 16:26:34.894621 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.894599    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:34.927551 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.927472    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:34.928416 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.928400    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:34.928497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.928435    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:34.928497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.928452    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:34.928497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.928490    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-221.ec2.internal"
Sep 15 16:26:34.934121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.934107    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-221.ec2.internal"
Sep 15 16:26:34.934193 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.934127    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-221.ec2.internal\": node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:34.950873 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:34.950855    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:34.993111 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.993076    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal"]
Sep 15 16:26:34.993293 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.993160    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:34.994133 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.994116    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:34.994192 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.994147    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:34.994192 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.994156    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:34.996537 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.996524    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:34.996671 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.996656    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:34.996712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.996687    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:34.997338 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.997322    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:34.997428 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.997351    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:34.997428 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.997360    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:34.997428 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.997323    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:34.997428 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.997424    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:34.997562 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.997437    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:34.999668 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.999650    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:34.999753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:34.999676    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:35.000343 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.000321    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:35.000408 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.000347    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:35.000408 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.000363    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:35.022509 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.022484    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-221.ec2.internal\" not found" node="ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.026703 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.026685    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-221.ec2.internal\" not found" node="ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.051424 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.051396    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:35.063902 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.063863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a2542e0dc51a480320ae6b40060e9dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal\" (UID: \"3a2542e0dc51a480320ae6b40060e9dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.063902 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.063908    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fe48e1c24e6156964623e1b724b0c0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-221.ec2.internal\" (UID: \"7fe48e1c24e6156964623e1b724b0c0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.064066 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.063925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a2542e0dc51a480320ae6b40060e9dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal\" (UID: \"3a2542e0dc51a480320ae6b40060e9dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.151838 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.151803    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:35.164092 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.164068    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a2542e0dc51a480320ae6b40060e9dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal\" (UID: \"3a2542e0dc51a480320ae6b40060e9dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.164159 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.164101    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a2542e0dc51a480320ae6b40060e9dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal\" (UID: \"3a2542e0dc51a480320ae6b40060e9dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.164159 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.164121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fe48e1c24e6156964623e1b724b0c0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-221.ec2.internal\" (UID: \"7fe48e1c24e6156964623e1b724b0c0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.164229 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.164159    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7fe48e1c24e6156964623e1b724b0c0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-221.ec2.internal\" (UID: \"7fe48e1c24e6156964623e1b724b0c0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.164229 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.164157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a2542e0dc51a480320ae6b40060e9dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal\" (UID: \"3a2542e0dc51a480320ae6b40060e9dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.164229 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.164192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a2542e0dc51a480320ae6b40060e9dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal\" (UID: \"3a2542e0dc51a480320ae6b40060e9dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.252543 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.252515    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:35.325025 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.324986    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.329004 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.328982    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.353001 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.352979    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:35.453502 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.453456    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:35.554054 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.553993    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-221.ec2.internal\" not found"
Sep 15 16:26:35.637659 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.637633    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:35.662554 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.662532    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.680547 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.680527    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 16:26:35.682552 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.682537    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal"
Sep 15 16:26:35.689031 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.689018    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 16:26:35.702207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.702186    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 16:26:35.702350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.702333    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 16:26:35.702397 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.702366    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 16:26:35.743517 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.743483    2564 apiserver.go:52] "Watching apiserver"
Sep 15 16:26:35.762259 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.762234    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 16:26:35.763814 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.763473    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 16:26:35.765212 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.765184    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal","openshift-multus/network-metrics-daemon-gt76w","openshift-network-diagnostics/network-check-target-db66d","openshift-network-operator/iptables-alerter-8kqvx","kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal","openshift-dns/node-resolver-rkjh7","openshift-image-registry/node-ca-s2cwp","openshift-multus/multus-additional-cni-plugins-h4rqc","openshift-multus/multus-p2j6z","openshift-ovn-kubernetes/ovnkube-node-jcbrc","kube-system/global-pull-secret-syncer-5tx7l","kube-system/konnectivity-agent-58fbn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z","openshift-cluster-node-tuning-operator/tuned-xtg5k"]
Sep 15 16:26:35.766443 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.766397    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 16:21:36 +0000 UTC" deadline="2028-07-02 17:06:36.164618226 +0000 UTC"
Sep 15 16:26:35.766518 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.766450    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15744h40m0.398171155s"
Sep 15 16:26:35.767883 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.767870    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:35.767957 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.767943    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:35.770182 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.770161    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:35.770265 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.770232    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:35.772812 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.772793    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.775287 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.775266    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 16:26:35.775396 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.775265    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jpgw4\""
Sep 15 16:26:35.775396 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.775301    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:26:35.775488 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.775468    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 16:26:35.775623 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.775609    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.775817 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.775802    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 16:26:35.777711 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.777693    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 16:26:35.777841 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.777697    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sqwsx\""
Sep 15 16:26:35.777841 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.777745    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 16:26:35.778661 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.778644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.780784 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.780764    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.781016 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.780993    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 16:26:35.781113 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.781045    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 16:26:35.781113 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.781090    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 16:26:35.781221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.781136    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tlx92\""
Sep 15 16:26:35.782873 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.782857    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 16:26:35.782970 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.782862    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 16:26:35.783125 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.783109    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.783201 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.783109    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 16:26:35.783258 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.783113    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bh879\""
Sep 15 16:26:35.783482 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.783463    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 16:26:35.783566 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.783498    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 16:26:35.785443 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.785427    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.785802 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.785629    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 16:26:35.785802 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.785688    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nqr2z\""
Sep 15 16:26:35.787604 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.787587    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.787687 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.787636    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:35.788053 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.788035    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 16:26:35.788130 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.788072    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mbvxs\""
Sep 15 16:26:35.788130 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.788076    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 16:26:35.788130 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.788083    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 16:26:35.788130 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.788117    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 16:26:35.788276 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.788093    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 16:26:35.788276 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.788184    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 16:26:35.789974 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.789864    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:26:35.791989 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.791970    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tbn2k\""
Sep 15 16:26:35.792079 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.791976    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 16:26:35.792139 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.792094    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 16:26:35.792271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.792254    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.794995 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.794650    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 16:26:35.794995 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.794767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.794995 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.794659    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-prlxs\""
Sep 15 16:26:35.794995 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.794829    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 16:26:35.794995 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.794840    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 16:26:35.794995 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.794973    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tp9zq"
Sep 15 16:26:35.797157 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.797140    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-v8g46\""
Sep 15 16:26:35.797237 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.797196    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:26:35.797905 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.797893    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 16:26:35.803268 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.803249    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tp9zq"
Sep 15 16:26:35.863476 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.863452    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 16:26:35.867995 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.867975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-env-overrides\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868107 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868001    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysctl-conf\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.868107 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868021    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-host\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.868107 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868037    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-cni-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.868107 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868058    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnv4\" (UniqueName: \"kubernetes.io/projected/21f87036-4cef-43d9-b366-cecbb41ea28f-kube-api-access-rqnv4\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.868107 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868077    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-slash\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868114    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-var-lib-kubelet\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47ltv\" (UniqueName: \"kubernetes.io/projected/b84e787d-7468-4967-963a-bdf15591987a-kube-api-access-47ltv\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868157    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-device-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868171    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-cni-bin\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868185    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovn-node-metrics-cert\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-hostroot\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-dbus\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868240    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a041ecec-9cc4-47e6-92c2-6e2213e5353f-konnectivity-ca\") pod \"konnectivity-agent-58fbn\" (UID: \"a041ecec-9cc4-47e6-92c2-6e2213e5353f\") " pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:26:35.868273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868268    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-socket-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868282    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-etc-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-ovn\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61a16b0b-3551-4b57-a1b5-116f6b194072-tmp\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47f4965d-054d-40a3-95d7-5771be4ae2e9-serviceca\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868343    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f87036-4cef-43d9-b366-cecbb41ea28f-iptables-alerter-script\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868376    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bd0a3f9-d122-4b8b-a617-38354c3e1687-hosts-file\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-run-netns\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-cni-netd\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868428    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzx7\" (UniqueName: \"kubernetes.io/projected/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-kube-api-access-sjzx7\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysconfig\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868468    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-multus-certs\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgspc\" (UniqueName: \"kubernetes.io/projected/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-kube-api-access-dgspc\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868514    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-kubernetes\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868544    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczkg\" (UniqueName: \"kubernetes.io/projected/61a16b0b-3551-4b57-a1b5-116f6b194072-kube-api-access-qczkg\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.868582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868570    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9pdw\" (UniqueName: \"kubernetes.io/projected/47f4965d-054d-40a3-95d7-5771be4ae2e9-kube-api-access-z9pdw\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-cni-multus\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868620    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-cnibin\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868636    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd0a3f9-d122-4b8b-a617-38354c3e1687-tmp-dir\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868685    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-netns\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47f4965d-054d-40a3-95d7-5771be4ae2e9-host\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868712    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-socket-dir-parent\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868725    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52976\" (UniqueName: \"kubernetes.io/projected/3bd0a3f9-d122-4b8b-a617-38354c3e1687-kube-api-access-52976\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868803    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-lib-modules\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-system-cni-dir\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868845    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.869148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-daemon-config\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868890    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-kubelet-config\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-run\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868953    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-cni-binary-copy\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868970    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovnkube-config\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovnkube-script-lib\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.868999    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869014    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-systemd\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-system-cni-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-k8s-cni-cncf-io\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869090    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm96f\" (UniqueName: \"kubernetes.io/projected/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-kube-api-access-qm96f\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-log-socket\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-modprobe-d\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869187    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysctl-d\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.869753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-sys\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869236    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-conf-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f87036-4cef-43d9-b366-cecbb41ea28f-host-slash\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-var-lib-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869303    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-node-log\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869348    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-systemd\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869362    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-tuned\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-cni-bin\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869421    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-etc-kubernetes\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2mj\" (UniqueName: \"kubernetes.io/projected/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-kube-api-access-sw2mj\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-systemd-units\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a041ecec-9cc4-47e6-92c2-6e2213e5353f-agent-certs\") pod \"konnectivity-agent-58fbn\" (UID: \"a041ecec-9cc4-47e6-92c2-6e2213e5353f\") " pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869512    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-sys-fs\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869543    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-kubelet\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-kubelet\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.870220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869581    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-cnibin\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.870831 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.869594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-registration-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.909863 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.909831    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:35.933093 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:35.933068    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a2542e0dc51a480320ae6b40060e9dd.slice/crio-9bfaddd2fe87cecbbadb5e36abef9a955687d7505b586a12518a8c04804bddb0 WatchSource:0}: Error finding container 9bfaddd2fe87cecbbadb5e36abef9a955687d7505b586a12518a8c04804bddb0: Status 404 returned error can't find the container with id 9bfaddd2fe87cecbbadb5e36abef9a955687d7505b586a12518a8c04804bddb0
Sep 15 16:26:35.939677 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.939657    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:26:35.970038 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-socket-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.970038 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-etc-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-ovn\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970068    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61a16b0b-3551-4b57-a1b5-116f6b194072-tmp\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970081    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47f4965d-054d-40a3-95d7-5771be4ae2e9-serviceca\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970119    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f87036-4cef-43d9-b366-cecbb41ea28f-iptables-alerter-script\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-etc-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bd0a3f9-d122-4b8b-a617-38354c3e1687-hosts-file\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970144    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-ovn\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970164    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-socket-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970190    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-run-netns\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3bd0a3f9-d122-4b8b-a617-38354c3e1687-hosts-file\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.970249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-cni-netd\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-run-netns\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzx7\" (UniqueName: \"kubernetes.io/projected/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-kube-api-access-sjzx7\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970295    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysconfig\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-multus-certs\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970325    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-cni-netd\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgspc\" (UniqueName: \"kubernetes.io/projected/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-kube-api-access-dgspc\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970367    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-kubernetes\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970373    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczkg\" (UniqueName: \"kubernetes.io/projected/61a16b0b-3551-4b57-a1b5-116f6b194072-kube-api-access-qczkg\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970419    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9pdw\" (UniqueName: \"kubernetes.io/projected/47f4965d-054d-40a3-95d7-5771be4ae2e9-kube-api-access-z9pdw\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-cni-multus\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970448    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysconfig\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-cnibin\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970572    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd0a3f9-d122-4b8b-a617-38354c3e1687-tmp-dir\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.970730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-netns\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47f4965d-054d-40a3-95d7-5771be4ae2e9-host\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-socket-dir-parent\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52976\" (UniqueName: \"kubernetes.io/projected/3bd0a3f9-d122-4b8b-a617-38354c3e1687-kube-api-access-52976\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-cni-multus\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970842    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-lib-modules\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-system-cni-dir\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/47f4965d-054d-40a3-95d7-5771be4ae2e9-serviceca\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970968    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-kubernetes\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970985    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f87036-4cef-43d9-b366-cecbb41ea28f-iptables-alerter-script\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.970945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-multus-certs\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971019    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971042    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-daemon-config\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971071    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-kubelet-config\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.971527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-cnibin\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-run\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-cni-binary-copy\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971170    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovnkube-config\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971193    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovnkube-script-lib\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971217    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971220    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bd0a3f9-d122-4b8b-a617-38354c3e1687-tmp-dir\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-systemd\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971329    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-system-cni-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971351    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-k8s-cni-cncf-io\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm96f\" (UniqueName: \"kubernetes.io/projected/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-kube-api-access-qm96f\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-log-socket\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.971403    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-modprobe-d\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysctl-d\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.972420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-sys\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971470    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-conf-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.971492    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs podName:3209aee8-cc28-4cc4-a3f0-c6af9ede9a60 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:36.471467985 +0000 UTC m=+2.073885607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs") pod "network-metrics-daemon-gt76w" (UID: "3209aee8-cc28-4cc4-a3f0-c6af9ede9a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971509    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-conf-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971549    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f87036-4cef-43d9-b366-cecbb41ea28f-host-slash\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-var-lib-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971593    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-system-cni-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971586    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-k8s-cni-cncf-io\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971623    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-log-socket\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-system-cni-dir\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-run-netns\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971720    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-socket-dir-parent\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971734    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f87036-4cef-43d9-b366-cecbb41ea28f-host-slash\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-daemon-config\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.973405 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47f4965d-054d-40a3-95d7-5771be4ae2e9-host\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-kubelet-config\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971782    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971832    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-node-log\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971844    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-sys\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-systemd\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971871    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysctl-d\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-tuned\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971955    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-node-log\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-run\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.971974    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.971984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-systemd\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-cni-bin\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.972037    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret podName:0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e nodeName:}" failed. No retries permitted until 2026-09-15 16:26:36.472019773 +0000 UTC m=+2.074437380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret") pod "global-pull-secret-syncer-5tx7l" (UID: "0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972069    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b84e787d-7468-4967-963a-bdf15591987a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.974238 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972077    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-var-lib-openvswitch\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972066    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-etc-kubernetes\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-modprobe-d\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972152    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-lib-modules\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2mj\" (UniqueName: \"kubernetes.io/projected/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-kube-api-access-sw2mj\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972232    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-cni-bin\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972271    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-etc-kubernetes\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-systemd-units\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972287    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-run-systemd\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972317    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-cni-binary-copy\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972361    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a041ecec-9cc4-47e6-92c2-6e2213e5353f-agent-certs\") pod \"konnectivity-agent-58fbn\" (UID: \"a041ecec-9cc4-47e6-92c2-6e2213e5353f\") " pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972433    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-sys-fs\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-kubelet\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-systemd-units\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972501    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-kubelet\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-cnibin\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.975041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-registration-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-env-overrides\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysctl-conf\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-host\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-cni-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972661    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnv4\" (UniqueName: \"kubernetes.io/projected/21f87036-4cef-43d9-b366-cecbb41ea28f-kube-api-access-rqnv4\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-slash\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-var-lib-kubelet\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47ltv\" (UniqueName: \"kubernetes.io/projected/b84e787d-7468-4967-963a-bdf15591987a-kube-api-access-47ltv\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972740    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovnkube-config\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972745    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-device-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972762    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovnkube-script-lib\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972773    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-device-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972791    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-cni-bin\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-slash\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovn-node-metrics-cert\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-hostroot\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.975644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972877    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-sysctl-conf\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972903    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-host\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-dbus\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972975    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b84e787d-7468-4967-963a-bdf15591987a-cnibin\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.972984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-multus-cni-dir\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-hostroot\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973017    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a041ecec-9cc4-47e6-92c2-6e2213e5353f-konnectivity-ca\") pod \"konnectivity-agent-58fbn\" (UID: \"a041ecec-9cc4-47e6-92c2-6e2213e5353f\") " pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-cni-bin\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973048    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61a16b0b-3551-4b57-a1b5-116f6b194072-var-lib-kubelet\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-registration-dir\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973088    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-host-kubelet\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973102    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-dbus\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-sys-fs\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973176    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-host-var-lib-kubelet\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973462    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-env-overrides\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.973858    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a041ecec-9cc4-47e6-92c2-6e2213e5353f-konnectivity-ca\") pod \"konnectivity-agent-58fbn\" (UID: \"a041ecec-9cc4-47e6-92c2-6e2213e5353f\") " pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.975394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-ovn-node-metrics-cert\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.975417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/61a16b0b-3551-4b57-a1b5-116f6b194072-etc-tuned\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.976128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.975451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61a16b0b-3551-4b57-a1b5-116f6b194072-tmp\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.976596 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.975587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a041ecec-9cc4-47e6-92c2-6e2213e5353f-agent-certs\") pod \"konnectivity-agent-58fbn\" (UID: \"a041ecec-9cc4-47e6-92c2-6e2213e5353f\") " pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:26:35.978036 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.978015    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:35.978144 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.978041    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:35.978144 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.978055    2564 projected.go:196] Error preparing data for projected volume kube-api-access-v9bgb for pod openshift-network-diagnostics/network-check-target-db66d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:35.978256 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:35.978157    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb podName:584a2e9a-89de-477a-9450-abdbc0d5b622 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:36.478139345 +0000 UTC m=+2.080556962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-v9bgb" (UniqueName: "kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb") pod "network-check-target-db66d" (UID: "584a2e9a-89de-477a-9450-abdbc0d5b622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:35.980183 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.980160    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzx7\" (UniqueName: \"kubernetes.io/projected/7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35-kube-api-access-sjzx7\") pod \"ovnkube-node-jcbrc\" (UID: \"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:35.980278 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.980197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9pdw\" (UniqueName: \"kubernetes.io/projected/47f4965d-054d-40a3-95d7-5771be4ae2e9-kube-api-access-z9pdw\") pod \"node-ca-s2cwp\" (UID: \"47f4965d-054d-40a3-95d7-5771be4ae2e9\") " pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:35.980362 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.980335    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczkg\" (UniqueName: \"kubernetes.io/projected/61a16b0b-3551-4b57-a1b5-116f6b194072-kube-api-access-qczkg\") pod \"tuned-xtg5k\" (UID: \"61a16b0b-3551-4b57-a1b5-116f6b194072\") " pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:35.980516 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.980501    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgspc\" (UniqueName: \"kubernetes.io/projected/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-kube-api-access-dgspc\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:35.982094 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.982077    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52976\" (UniqueName: \"kubernetes.io/projected/3bd0a3f9-d122-4b8b-a617-38354c3e1687-kube-api-access-52976\") pod \"node-resolver-rkjh7\" (UID: \"3bd0a3f9-d122-4b8b-a617-38354c3e1687\") " pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:35.982313 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.982292    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2mj\" (UniqueName: \"kubernetes.io/projected/0ac161c6-7af9-4c0f-9697-2ae5a38b377f-kube-api-access-sw2mj\") pod \"multus-p2j6z\" (UID: \"0ac161c6-7af9-4c0f-9697-2ae5a38b377f\") " pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:35.982646 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.982632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnv4\" (UniqueName: \"kubernetes.io/projected/21f87036-4cef-43d9-b366-cecbb41ea28f-kube-api-access-rqnv4\") pod \"iptables-alerter-8kqvx\" (UID: \"21f87036-4cef-43d9-b366-cecbb41ea28f\") " pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:35.982964 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:35.982946    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm96f\" (UniqueName: \"kubernetes.io/projected/23aeb6e4-5d2d-49e9-addf-4fdd192a567e-kube-api-access-qm96f\") pod \"aws-ebs-csi-driver-node-zxc4z\" (UID: \"23aeb6e4-5d2d-49e9-addf-4fdd192a567e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:37.254829 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.254812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47ltv\" (UniqueName: \"kubernetes.io/projected/b84e787d-7468-4967-963a-bdf15591987a-kube-api-access-47ltv\") pod \"multus-additional-cni-plugins-h4rqc\" (UID: \"b84e787d-7468-4967-963a-bdf15591987a\") " pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:37.324224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.324198    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:37.375417 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.375331    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8kqvx"
Sep 15 16:26:37.386247 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.386211    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f87036_4cef_43d9_b366_cecbb41ea28f.slice/crio-57cd9fef1658b372387b5c4a52fb82a3e1ee2638fd0a3765b8eb110e510b6d22 WatchSource:0}: Error finding container 57cd9fef1658b372387b5c4a52fb82a3e1ee2638fd0a3765b8eb110e510b6d22: Status 404 returned error can't find the container with id 57cd9fef1658b372387b5c4a52fb82a3e1ee2638fd0a3765b8eb110e510b6d22
Sep 15 16:26:37.390387 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.390369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rkjh7"
Sep 15 16:26:37.395311 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.395294    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s2cwp"
Sep 15 16:26:37.399931 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.399910    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd0a3f9_d122_4b8b_a617_38354c3e1687.slice/crio-4bc69c3f8de696022778ae5a853cbe77573bb78e569477815bbcdb0c3073e1e7 WatchSource:0}: Error finding container 4bc69c3f8de696022778ae5a853cbe77573bb78e569477815bbcdb0c3073e1e7: Status 404 returned error can't find the container with id 4bc69c3f8de696022778ae5a853cbe77573bb78e569477815bbcdb0c3073e1e7
Sep 15 16:26:37.404172 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.404153    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f4965d_054d_40a3_95d7_5771be4ae2e9.slice/crio-285c2375e2f388e386f6efe19389ee47d6d0f9929b44b2894703987c6a1996ca WatchSource:0}: Error finding container 285c2375e2f388e386f6efe19389ee47d6d0f9929b44b2894703987c6a1996ca: Status 404 returned error can't find the container with id 285c2375e2f388e386f6efe19389ee47d6d0f9929b44b2894703987c6a1996ca
Sep 15 16:26:37.424274 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.424248    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h4rqc"
Sep 15 16:26:37.435434 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.435412    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84e787d_7468_4967_963a_bdf15591987a.slice/crio-00e1280500396b662aaafa713ea8cb53c491fcb8226833d879f1de0b5410f362 WatchSource:0}: Error finding container 00e1280500396b662aaafa713ea8cb53c491fcb8226833d879f1de0b5410f362: Status 404 returned error can't find the container with id 00e1280500396b662aaafa713ea8cb53c491fcb8226833d879f1de0b5410f362
Sep 15 16:26:37.442567 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.442539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p2j6z"
Sep 15 16:26:37.450233 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.450213    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac161c6_7af9_4c0f_9697_2ae5a38b377f.slice/crio-8c057ea89150b1fd014a78a78e4dab221f9d926e574490d1d643e1b0ad7870c7 WatchSource:0}: Error finding container 8c057ea89150b1fd014a78a78e4dab221f9d926e574490d1d643e1b0ad7870c7: Status 404 returned error can't find the container with id 8c057ea89150b1fd014a78a78e4dab221f9d926e574490d1d643e1b0ad7870c7
Sep 15 16:26:37.459475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.459457    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:26:37.465311 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.465291    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:26:37.467475 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.467454    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f127dd2_3cd7_4868_b39a_c9aeaaa2ac35.slice/crio-814298af1177473d7b7ef9201842b6a541d12383e2430bf7ce44c3f5d5c36e59 WatchSource:0}: Error finding container 814298af1177473d7b7ef9201842b6a541d12383e2430bf7ce44c3f5d5c36e59: Status 404 returned error can't find the container with id 814298af1177473d7b7ef9201842b6a541d12383e2430bf7ce44c3f5d5c36e59
Sep 15 16:26:37.474059 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.474039    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda041ecec_9cc4_47e6_92c2_6e2213e5353f.slice/crio-d3b0d325e2f76636090fe5384a240317dde5c829923db30fd0743b43824f7d66 WatchSource:0}: Error finding container d3b0d325e2f76636090fe5384a240317dde5c829923db30fd0743b43824f7d66: Status 404 returned error can't find the container with id d3b0d325e2f76636090fe5384a240317dde5c829923db30fd0743b43824f7d66
Sep 15 16:26:37.497200 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.497175    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z"
Sep 15 16:26:37.506448 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.506430    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xtg5k"
Sep 15 16:26:37.506605 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.506585    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23aeb6e4_5d2d_49e9_addf_4fdd192a567e.slice/crio-c6119cb8b73e7536ac3aa1185eb530a8ff542a5b19d866d6ee8e9bd299a28525 WatchSource:0}: Error finding container c6119cb8b73e7536ac3aa1185eb530a8ff542a5b19d866d6ee8e9bd299a28525: Status 404 returned error can't find the container with id c6119cb8b73e7536ac3aa1185eb530a8ff542a5b19d866d6ee8e9bd299a28525
Sep 15 16:26:37.514339 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:37.514319    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a16b0b_3551_4b57_a1b5_116f6b194072.slice/crio-33555b845edcc1050c603b271ef892363244818295f20404271345f5f1ed00e8 WatchSource:0}: Error finding container 33555b845edcc1050c603b271ef892363244818295f20404271345f5f1ed00e8: Status 404 returned error can't find the container with id 33555b845edcc1050c603b271ef892363244818295f20404271345f5f1ed00e8
Sep 15 16:26:37.742832 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.742750    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:37.742832 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.742788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:37.743033 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:37.742888    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:37.743033 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:37.742897    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:37.743033 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:37.742957    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret podName:0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e nodeName:}" failed. No retries permitted until 2026-09-15 16:26:38.742939408 +0000 UTC m=+3.078894318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret") pod "global-pull-secret-syncer-5tx7l" (UID: "0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:37.743033 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:37.742973    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs podName:3209aee8-cc28-4cc4-a3f0-c6af9ede9a60 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:38.742966454 +0000 UTC m=+3.078921363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs") pod "network-metrics-daemon-gt76w" (UID: "3209aee8-cc28-4cc4-a3f0-c6af9ede9a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:37.843926 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.843899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:37.844100 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:37.844083    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:37.844165 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:37.844105    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:37.844165 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:37.844117    2564 projected.go:196] Error preparing data for projected volume kube-api-access-v9bgb for pod openshift-network-diagnostics/network-check-target-db66d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:37.844264 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:37.844180    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb podName:584a2e9a-89de-477a-9450-abdbc0d5b622 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:38.844162082 +0000 UTC m=+3.180116992 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9bgb" (UniqueName: "kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb") pod "network-check-target-db66d" (UID: "584a2e9a-89de-477a-9450-abdbc0d5b622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:37.877710 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:37.877676    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:38.072588 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.072483    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:21:37 +0000 UTC" deadline="2028-02-17 09:04:12.667216751 +0000 UTC"
Sep 15 16:26:38.072588 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.072531    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12472h37m34.594690459s"
Sep 15 16:26:38.182640 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.182578    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xtg5k" event={"ID":"61a16b0b-3551-4b57-a1b5-116f6b194072","Type":"ContainerStarted","Data":"33555b845edcc1050c603b271ef892363244818295f20404271345f5f1ed00e8"}
Sep 15 16:26:38.189016 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.188978    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" event={"ID":"b84e787d-7468-4967-963a-bdf15591987a","Type":"ContainerStarted","Data":"00e1280500396b662aaafa713ea8cb53c491fcb8226833d879f1de0b5410f362"}
Sep 15 16:26:38.201668 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.201638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kqvx" event={"ID":"21f87036-4cef-43d9-b366-cecbb41ea28f","Type":"ContainerStarted","Data":"57cd9fef1658b372387b5c4a52fb82a3e1ee2638fd0a3765b8eb110e510b6d22"}
Sep 15 16:26:38.209674 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.209640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z" event={"ID":"23aeb6e4-5d2d-49e9-addf-4fdd192a567e","Type":"ContainerStarted","Data":"c6119cb8b73e7536ac3aa1185eb530a8ff542a5b19d866d6ee8e9bd299a28525"}
Sep 15 16:26:38.216213 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.216174    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-58fbn" event={"ID":"a041ecec-9cc4-47e6-92c2-6e2213e5353f","Type":"ContainerStarted","Data":"d3b0d325e2f76636090fe5384a240317dde5c829923db30fd0743b43824f7d66"}
Sep 15 16:26:38.220736 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.220706    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"814298af1177473d7b7ef9201842b6a541d12383e2430bf7ce44c3f5d5c36e59"}
Sep 15 16:26:38.224769 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.224726    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2j6z" event={"ID":"0ac161c6-7af9-4c0f-9697-2ae5a38b377f","Type":"ContainerStarted","Data":"8c057ea89150b1fd014a78a78e4dab221f9d926e574490d1d643e1b0ad7870c7"}
Sep 15 16:26:38.232884 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.232859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s2cwp" event={"ID":"47f4965d-054d-40a3-95d7-5771be4ae2e9","Type":"ContainerStarted","Data":"285c2375e2f388e386f6efe19389ee47d6d0f9929b44b2894703987c6a1996ca"}
Sep 15 16:26:38.243301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.243274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rkjh7" event={"ID":"3bd0a3f9-d122-4b8b-a617-38354c3e1687","Type":"ContainerStarted","Data":"4bc69c3f8de696022778ae5a853cbe77573bb78e569477815bbcdb0c3073e1e7"}
Sep 15 16:26:38.245262 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.245224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal" event={"ID":"3a2542e0dc51a480320ae6b40060e9dd","Type":"ContainerStarted","Data":"9bfaddd2fe87cecbbadb5e36abef9a955687d7505b586a12518a8c04804bddb0"}
Sep 15 16:26:38.542538 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:26:38.539496    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe48e1c24e6156964623e1b724b0c0d.slice/crio-81cf228d9f5fc35311e245961537b8620691bfbdc0cad399fe64220c614ab1d8 WatchSource:0}: Error finding container 81cf228d9f5fc35311e245961537b8620691bfbdc0cad399fe64220c614ab1d8: Status 404 returned error can't find the container with id 81cf228d9f5fc35311e245961537b8620691bfbdc0cad399fe64220c614ab1d8
Sep 15 16:26:38.712215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.712185    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:38.753233 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.752483    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:38.753233 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.752582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:38.753233 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:38.752706    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:38.753233 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:38.752767    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret podName:0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e nodeName:}" failed. No retries permitted until 2026-09-15 16:26:40.752749801 +0000 UTC m=+5.088704715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret") pod "global-pull-secret-syncer-5tx7l" (UID: "0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:38.753233 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:38.753159    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:38.753233 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:38.753205    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs podName:3209aee8-cc28-4cc4-a3f0-c6af9ede9a60 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:40.753191013 +0000 UTC m=+5.089145926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs") pod "network-metrics-daemon-gt76w" (UID: "3209aee8-cc28-4cc4-a3f0-c6af9ede9a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:38.853130 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:38.853045    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:38.853357 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:38.853215    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:38.853357 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:38.853243    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:38.853357 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:38.853256    2564 projected.go:196] Error preparing data for projected volume kube-api-access-v9bgb for pod openshift-network-diagnostics/network-check-target-db66d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:38.853357 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:38.853316    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb podName:584a2e9a-89de-477a-9450-abdbc0d5b622 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:40.853298218 +0000 UTC m=+5.189253147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9bgb" (UniqueName: "kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb") pod "network-check-target-db66d" (UID: "584a2e9a-89de-477a-9450-abdbc0d5b622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:39.073040 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:39.072976    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:21:37 +0000 UTC" deadline="2028-03-30 20:28:33.625900828 +0000 UTC"
Sep 15 16:26:39.073040 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:39.073039    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13492h1m54.552866185s"
Sep 15 16:26:39.160693 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:39.160249    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:39.160693 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:39.160382    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:39.160893 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:39.160807    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:39.160945 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:39.160897    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:39.161003 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:39.160964    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:39.161056 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:39.161039    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:39.277215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:39.277142    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal" event={"ID":"7fe48e1c24e6156964623e1b724b0c0d","Type":"ContainerStarted","Data":"81cf228d9f5fc35311e245961537b8620691bfbdc0cad399fe64220c614ab1d8"}
Sep 15 16:26:40.771295 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:40.770586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:40.771295 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:40.770635    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:40.771295 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:40.770760    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:40.771295 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:40.770820    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs podName:3209aee8-cc28-4cc4-a3f0-c6af9ede9a60 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:44.770802283 +0000 UTC m=+9.106757198 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs") pod "network-metrics-daemon-gt76w" (UID: "3209aee8-cc28-4cc4-a3f0-c6af9ede9a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:40.771295 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:40.771200    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:40.771295 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:40.771246    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret podName:0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e nodeName:}" failed. No retries permitted until 2026-09-15 16:26:44.771232654 +0000 UTC m=+9.107187567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret") pod "global-pull-secret-syncer-5tx7l" (UID: "0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:40.872158 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:40.871571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:40.872158 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:40.871741    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:40.872158 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:40.871760    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:40.872158 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:40.871772    2564 projected.go:196] Error preparing data for projected volume kube-api-access-v9bgb for pod openshift-network-diagnostics/network-check-target-db66d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:40.872158 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:40.871827    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb podName:584a2e9a-89de-477a-9450-abdbc0d5b622 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:44.871811036 +0000 UTC m=+9.207765969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9bgb" (UniqueName: "kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb") pod "network-check-target-db66d" (UID: "584a2e9a-89de-477a-9450-abdbc0d5b622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:41.159158 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:41.159077    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:41.159311 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:41.159204    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:41.159311 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:41.159213    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:41.159447 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:41.159348    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:41.159447 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:41.159416    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:41.159521 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:41.159499    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:43.159946 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:43.159907    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:43.160369 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:43.159925    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:43.160369 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:43.160048    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:43.160369 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:43.159925    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:43.160369 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:43.160129    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:43.160369 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:43.160175    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:44.802548 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:44.802511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:44.802937 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:44.802566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:44.802937 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:44.802679    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:44.802937 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:44.802731    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs podName:3209aee8-cc28-4cc4-a3f0-c6af9ede9a60 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.802716881 +0000 UTC m=+17.138671803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs") pod "network-metrics-daemon-gt76w" (UID: "3209aee8-cc28-4cc4-a3f0-c6af9ede9a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:44.803073 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:44.803045    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:44.803122 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:44.803080    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret podName:0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.80307035 +0000 UTC m=+17.139025272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret") pod "global-pull-secret-syncer-5tx7l" (UID: "0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:44.903205 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:44.903119    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:44.903386 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:44.903313    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:44.903386 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:44.903348    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:44.903386 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:44.903361    2564 projected.go:196] Error preparing data for projected volume kube-api-access-v9bgb for pod openshift-network-diagnostics/network-check-target-db66d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:44.903551 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:44.903445    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb podName:584a2e9a-89de-477a-9450-abdbc0d5b622 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:52.903425103 +0000 UTC m=+17.239380029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9bgb" (UniqueName: "kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb") pod "network-check-target-db66d" (UID: "584a2e9a-89de-477a-9450-abdbc0d5b622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:45.158968 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:45.158884    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:45.158968 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:45.158915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:45.159156 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:45.158889    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:45.159156 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:45.159015    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:45.159156 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:45.159097    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:45.159258 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:45.159164    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:47.159938 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:47.159848    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:47.160536 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:47.159861    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:47.160536 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:47.159990    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:47.160536 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:47.159998    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:47.160536 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:47.160057    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:47.160536 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:47.160182    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:49.158868 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:49.158836    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:49.159291 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:49.158879    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:49.159291 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:49.158879    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:49.159291 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:49.158960    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:49.159291 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:49.159055    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:49.159291 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:49.159129    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:51.159470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:51.159440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:51.159912 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:51.159440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:51.159912 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:51.159566    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:51.159912 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:51.159650    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:51.159912 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:51.159440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:51.159912 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:51.159736    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:52.865635 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:52.865597    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:52.866093 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:52.865648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:52.866093 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:52.865763    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:52.866093 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:52.865768    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:52.866093 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:52.865826    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs podName:3209aee8-cc28-4cc4-a3f0-c6af9ede9a60 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:08.865809393 +0000 UTC m=+33.201764306 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs") pod "network-metrics-daemon-gt76w" (UID: "3209aee8-cc28-4cc4-a3f0-c6af9ede9a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:52.866093 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:52.865840    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret podName:0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:08.865834188 +0000 UTC m=+33.201789097 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret") pod "global-pull-secret-syncer-5tx7l" (UID: "0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:52.966486 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:52.966449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:52.966686 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:52.966604    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:52.966686 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:52.966629    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:52.966686 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:52.966642    2564 projected.go:196] Error preparing data for projected volume kube-api-access-v9bgb for pod openshift-network-diagnostics/network-check-target-db66d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:52.966841 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:52.966710    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb podName:584a2e9a-89de-477a-9450-abdbc0d5b622 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:08.966690738 +0000 UTC m=+33.302645650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9bgb" (UniqueName: "kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb") pod "network-check-target-db66d" (UID: "584a2e9a-89de-477a-9450-abdbc0d5b622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:53.159616 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:53.159546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:53.159864 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:53.159663    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:53.159864 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:53.159676    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:53.159864 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:53.159710    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:53.159864 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:53.159806    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:53.160070 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:53.159897    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:55.159210 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:55.159173    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:55.159210 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:55.159200    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:55.159705 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:55.159201    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:55.159705 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:55.159299    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:55.159705 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:55.159413    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:55.159705 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:55.159457    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:57.159049 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:57.159010    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:57.159464 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:57.159067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:57.159464 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:57.159160    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:57.159464 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:57.159248    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:57.159464 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:57.159277    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:57.159464 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:57.159336    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:58.317542 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.317132    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z" event={"ID":"23aeb6e4-5d2d-49e9-addf-4fdd192a567e","Type":"ContainerStarted","Data":"0700b8b42f49abb6d1fa60b9ce39683136b98f320bb8a2ff1513153a29719997"}
Sep 15 16:26:58.319103 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.319056    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-58fbn" event={"ID":"a041ecec-9cc4-47e6-92c2-6e2213e5353f","Type":"ContainerStarted","Data":"56ca4f3730d6b01cb52e12699d05e961f7152ea91bb304e93b452b69882c5652"}
Sep 15 16:26:58.322775 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.322748    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"ddfd6e54b93b2a0a905caa2f9588d8d6a4f3ffda8808f9fb86513f711b18164b"}
Sep 15 16:26:58.322878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.322826    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"3f428c16686a97504a271eaeb86993eb806c769c284b234356c4c8ceeebc5840"}
Sep 15 16:26:58.322938 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.322877    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"a613d912470eb2da7b27218038af09e4c2d5a943d3262a8291054988f58fe8d3"}
Sep 15 16:26:58.323010 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.322982    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"817fdeaf7e72df5b4da2c9f9b191b4a0e4cb2049da5c9cd22aa313c1989bdfa9"}
Sep 15 16:26:58.323121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.323034    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"e98e1775f97bf8befdb33ceeb876ad1ae61fbc53a1fa981193112d7d7021c841"}
Sep 15 16:26:58.323121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.323076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"cd28d7851ca5f8808e76d1670f5000bd0b5c875005c61229f82dbf18b6864486"}
Sep 15 16:26:58.324472 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.324444    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p2j6z" event={"ID":"0ac161c6-7af9-4c0f-9697-2ae5a38b377f","Type":"ContainerStarted","Data":"093df1bbf95220bc4967840ab12410849564b6efbafdcc95b3d486243243a5be"}
Sep 15 16:26:58.326062 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.326038    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s2cwp" event={"ID":"47f4965d-054d-40a3-95d7-5771be4ae2e9","Type":"ContainerStarted","Data":"a9db0c10463140b25aa7eccbe1a0fcee873de5acd9eed073ca1e10af344669e6"}
Sep 15 16:26:58.327570 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.327547    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rkjh7" event={"ID":"3bd0a3f9-d122-4b8b-a617-38354c3e1687","Type":"ContainerStarted","Data":"301d08873b1723fa7ac1da6e29709725a8bf0ca9081c803bbd634ddaeb40c8b2"}
Sep 15 16:26:58.329227 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.329194    2564 generic.go:356] "Generic (PLEG): container finished" podID="3a2542e0dc51a480320ae6b40060e9dd" containerID="70a38161dfbd1be310dc4db025d51af3e2b02222a3062630f8a38da315b96dad" exitCode=0
Sep 15 16:26:58.329321 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.329271    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal" event={"ID":"3a2542e0dc51a480320ae6b40060e9dd","Type":"ContainerDied","Data":"70a38161dfbd1be310dc4db025d51af3e2b02222a3062630f8a38da315b96dad"}
Sep 15 16:26:58.330846 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.330823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xtg5k" event={"ID":"61a16b0b-3551-4b57-a1b5-116f6b194072","Type":"ContainerStarted","Data":"58f12fff54580f00f6a0d2528e798ee698e59d7bc07e067eeade1bee75881066"}
Sep 15 16:26:58.332741 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.332718    2564 generic.go:356] "Generic (PLEG): container finished" podID="b84e787d-7468-4967-963a-bdf15591987a" containerID="ca665c5e1ecc0ccb37c5bba9c2d7ff5ff840bfd0f0bf1cdd45c877ed3998a6d6" exitCode=0
Sep 15 16:26:58.332840 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.332814    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" event={"ID":"b84e787d-7468-4967-963a-bdf15591987a","Type":"ContainerDied","Data":"ca665c5e1ecc0ccb37c5bba9c2d7ff5ff840bfd0f0bf1cdd45c877ed3998a6d6"}
Sep 15 16:26:58.334525 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.334504    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal" event={"ID":"7fe48e1c24e6156964623e1b724b0c0d","Type":"ContainerStarted","Data":"e9461a78d17afde1dba370d3083c0c46fae19756bb655d9e5d9b527a7fb74832"}
Sep 15 16:26:58.360454 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.360336    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-58fbn" podStartSLOduration=10.053540788 podStartE2EDuration="22.360316671s" totalImagesPullingTime="12.306775883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:58.341054314 +0000 UTC m=+22.677009246" watchObservedRunningTime="2026-09-15 16:26:58.360316671 +0000 UTC m=+22.696271603"
Sep 15 16:26:58.360880 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.360848    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-221.ec2.internal" podStartSLOduration=22.360837852 podStartE2EDuration="22.360837852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:58.360071777 +0000 UTC m=+22.696026711" watchObservedRunningTime="2026-09-15 16:26:58.360837852 +0000 UTC m=+22.696792784"
Sep 15 16:26:58.376361 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.376306    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s2cwp" podStartSLOduration=2.5184463470000003 podStartE2EDuration="22.376288878s" totalImagesPullingTime="19.857842531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:58.375853828 +0000 UTC m=+22.711808760" watchObservedRunningTime="2026-09-15 16:26:58.376288878 +0000 UTC m=+22.712243810"
Sep 15 16:26:58.395322 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.395270    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p2j6z" podStartSLOduration=2.374244274 podStartE2EDuration="22.395254982s" totalImagesPullingTime="20.021010708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:58.395110684 +0000 UTC m=+22.731065615" watchObservedRunningTime="2026-09-15 16:26:58.395254982 +0000 UTC m=+22.731209913"
Sep 15 16:26:58.416843 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.416801    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xtg5k" podStartSLOduration=2.6455870729999997 podStartE2EDuration="22.416784834s" totalImagesPullingTime="19.771197761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:58.416700848 +0000 UTC m=+22.752655780" watchObservedRunningTime="2026-09-15 16:26:58.416784834 +0000 UTC m=+22.752739767"
Sep 15 16:26:58.489211 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:58.489162    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rkjh7" podStartSLOduration=6.377001158 podStartE2EDuration="22.489146486s" totalImagesPullingTime="16.112145328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:58.459199584 +0000 UTC m=+22.795154515" watchObservedRunningTime="2026-09-15 16:26:58.489146486 +0000 UTC m=+22.825101394"
Sep 15 16:26:59.159289 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.159261    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:26:59.159289 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.159280    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:26:59.159490 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:59.159418    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:26:59.159490 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.159454    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:26:59.159595 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:59.159532    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:26:59.159595 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:26:59.159580    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:26:59.228960 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.228754    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 16:26:59.338052 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.337974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal" event={"ID":"3a2542e0dc51a480320ae6b40060e9dd","Type":"ContainerStarted","Data":"e2e9c6fd7b9e915ce065da819238ac70e5a7bc21711f598b0e7de70e4d4ee8d7"}
Sep 15 16:26:59.339564 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.339541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8kqvx" event={"ID":"21f87036-4cef-43d9-b366-cecbb41ea28f","Type":"ContainerStarted","Data":"5ca718b4ebac0249eda07675afe555abda607e31d5072cdbefbb60f7ecea5b0e"}
Sep 15 16:26:59.341271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.341243    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z" event={"ID":"23aeb6e4-5d2d-49e9-addf-4fdd192a567e","Type":"ContainerStarted","Data":"ccd4432f0d116e41baeba1d39183ac6314d0aad69537688b01b4b66157498bb9"}
Sep 15 16:26:59.372688 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.372638    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-221.ec2.internal" podStartSLOduration=23.372624547 podStartE2EDuration="23.372624547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:59.355530579 +0000 UTC m=+23.691485510" watchObservedRunningTime="2026-09-15 16:26:59.372624547 +0000 UTC m=+23.708579477"
Sep 15 16:26:59.372828 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:26:59.372771    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8kqvx" podStartSLOduration=3.497175175 podStartE2EDuration="23.372766507s" totalImagesPullingTime="19.875591332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:59.372551799 +0000 UTC m=+23.708506730" watchObservedRunningTime="2026-09-15 16:26:59.372766507 +0000 UTC m=+23.708721435"
Sep 15 16:27:00.105915 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:00.105790    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T16:26:59.228958541Z","UUID":"dfed3f33-2771-498c-b44d-ce9f01c27aa0","Handler":null,"Name":"","Endpoint":""}
Sep 15 16:27:00.107648 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:00.107628    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 16:27:00.107776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:00.107658    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 16:27:00.347981 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:00.347946    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"98b1d3ac4a6afc326cdcba78decdbe853ea8ab910af8fef2508cd2acfe9eac13"}
Sep 15 16:27:00.574373 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:00.574339    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:27:00.575249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:00.575230    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:27:01.159539 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:01.159125    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:01.159539 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:01.159180    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:01.159539 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:01.159234    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:01.159539 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:01.159293    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:01.159539 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:01.159358    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:01.159539 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:01.159468    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:03.159050 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:03.159020    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:03.159050 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:03.159050    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:03.159554 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:03.159053    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:03.159554 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:03.159141    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:03.159554 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:03.159234    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:03.159554 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:03.159328    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:04.361577 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.361297    2564 generic.go:356] "Generic (PLEG): container finished" podID="b84e787d-7468-4967-963a-bdf15591987a" containerID="09c557a6e28701d21d38dd3fbb08e822ed3aa40bca5bc0a1fc206294e353c81b" exitCode=0
Sep 15 16:27:04.362221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.361384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" event={"ID":"b84e787d-7468-4967-963a-bdf15591987a","Type":"ContainerDied","Data":"09c557a6e28701d21d38dd3fbb08e822ed3aa40bca5bc0a1fc206294e353c81b"}
Sep 15 16:27:04.364131 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.364101    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z" event={"ID":"23aeb6e4-5d2d-49e9-addf-4fdd192a567e","Type":"ContainerStarted","Data":"ed5c13b6635325b095cf45d9cab589fb953a5ac310c840da89af608ea03d1059"}
Sep 15 16:27:04.367950 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.367929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" event={"ID":"7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35","Type":"ContainerStarted","Data":"740a3901264f5604baa9c8817b1cb0f482a4c599b4803b9b3ff39264321b20d2"}
Sep 15 16:27:04.368156 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.368135    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:27:04.368267 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.368200    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:27:04.384021 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.383992    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:27:04.384446 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.384428    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:27:04.395966 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.395912    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxc4z" podStartSLOduration=12.295068874 podStartE2EDuration="28.395894815s" totalImagesPullingTime="16.100825941s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:04.395531657 +0000 UTC m=+28.731486565" watchObservedRunningTime="2026-09-15 16:27:04.395894815 +0000 UTC m=+28.731849747"
Sep 15 16:27:04.421752 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:04.421708    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc" podStartSLOduration=8.566800162 podStartE2EDuration="28.421694833s" totalImagesPullingTime="19.854894671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:04.421615323 +0000 UTC m=+28.757570268" watchObservedRunningTime="2026-09-15 16:27:04.421694833 +0000 UTC m=+28.757649763"
Sep 15 16:27:05.141762 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:05.141727    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:27:05.141935 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:05.141892    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 16:27:05.142369 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:05.142342    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-58fbn"
Sep 15 16:27:05.159185 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:05.159158    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:05.159321 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:05.159238    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:05.159321 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:05.159250    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:05.159463 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:05.159356    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:05.159463 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:05.159441    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:05.159576 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:05.159534    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:05.370385 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:05.370358    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 16:27:06.373928 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:06.373892    2564 generic.go:356] "Generic (PLEG): container finished" podID="b84e787d-7468-4967-963a-bdf15591987a" containerID="a3b415a3629474e90eec4b293e0573b216aa8d260eb8c73aa030737f4df571dd" exitCode=0
Sep 15 16:27:06.374435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:06.373974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" event={"ID":"b84e787d-7468-4967-963a-bdf15591987a","Type":"ContainerDied","Data":"a3b415a3629474e90eec4b293e0573b216aa8d260eb8c73aa030737f4df571dd"}
Sep 15 16:27:06.374435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:06.374057    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 16:27:06.642629 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:06.642534    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:27:07.159127 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:07.159088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:07.159301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:07.159091    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:07.159301 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:07.159188    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:07.159301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:07.159091    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:07.159301 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:07.159279    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:07.159489 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:07.159348    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:08.872615 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:08.872569    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:08.872615 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:08.872614    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:08.873084 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:08.872721    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:08.873084 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:08.872721    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:08.873084 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:08.872784    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs podName:3209aee8-cc28-4cc4-a3f0-c6af9ede9a60 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:40.872769147 +0000 UTC m=+65.208724075 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs") pod "network-metrics-daemon-gt76w" (UID: "3209aee8-cc28-4cc4-a3f0-c6af9ede9a60") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:08.873084 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:08.872798    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret podName:0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:40.87279289 +0000 UTC m=+65.208747799 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret") pod "global-pull-secret-syncer-5tx7l" (UID: "0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:08.973843 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:08.973748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:08.973999 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:08.973862    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:08.973999 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:08.973877    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:08.973999 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:08.973885    2564 projected.go:196] Error preparing data for projected volume kube-api-access-v9bgb for pod openshift-network-diagnostics/network-check-target-db66d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:08.973999 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:08.973936    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb podName:584a2e9a-89de-477a-9450-abdbc0d5b622 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:40.973923669 +0000 UTC m=+65.309878577 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-v9bgb" (UniqueName: "kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb") pod "network-check-target-db66d" (UID: "584a2e9a-89de-477a-9450-abdbc0d5b622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:09.159457 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:09.159423    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:09.159457 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:09.159461    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:09.159648 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:09.159510    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:09.159648 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:09.159525    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:09.159648 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:09.159588    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:09.159648 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:09.159629    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:09.382413 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:09.382363    2564 generic.go:356] "Generic (PLEG): container finished" podID="b84e787d-7468-4967-963a-bdf15591987a" containerID="a9b8de27cfe176c86f92658eb46eaa106f111ad4274ba5bb760c009210127176" exitCode=0
Sep 15 16:27:09.382413 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:09.382416    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" event={"ID":"b84e787d-7468-4967-963a-bdf15591987a","Type":"ContainerDied","Data":"a9b8de27cfe176c86f92658eb46eaa106f111ad4274ba5bb760c009210127176"}
Sep 15 16:27:11.159543 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:11.159506    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:11.159543 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:11.159533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:11.160089 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:11.159516    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:11.160089 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:11.159620    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:11.160089 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:11.159700    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:11.160089 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:11.159765    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:13.159247 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:13.159209    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:13.159709 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:13.159255    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:13.159709 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:13.159311    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:13.159709 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:13.159327    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:13.159709 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:13.159386    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:13.159709 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:13.159485    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:15.159318 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:15.158832    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:15.159318 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:15.158965    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:15.159318 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:15.158973    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:15.159318 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:15.159002    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:15.159318 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:15.159092    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:15.159318 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:15.159186    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:16.401501 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:16.401470    2564 generic.go:356] "Generic (PLEG): container finished" podID="b84e787d-7468-4967-963a-bdf15591987a" containerID="4bf2d1b29b247772c62215c48802ffe0c74e448e8daf68b0139bfcb3d15b6b0d" exitCode=0
Sep 15 16:27:16.401905 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:16.401531    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" event={"ID":"b84e787d-7468-4967-963a-bdf15591987a","Type":"ContainerDied","Data":"4bf2d1b29b247772c62215c48802ffe0c74e448e8daf68b0139bfcb3d15b6b0d"}
Sep 15 16:27:17.159055 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:17.158861    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:17.159206 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:17.158893    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:17.159206 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:17.159140    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:17.159206 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:17.158944    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:17.159322 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:17.159203    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:17.159322 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:17.159279    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:17.407068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:17.407029    2564 generic.go:356] "Generic (PLEG): container finished" podID="b84e787d-7468-4967-963a-bdf15591987a" containerID="1e11619dbf1253256104ddb02be23e7b4eda0765da875574c1b6b3483c140023" exitCode=0
Sep 15 16:27:17.407440 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:17.407088    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" event={"ID":"b84e787d-7468-4967-963a-bdf15591987a","Type":"ContainerDied","Data":"1e11619dbf1253256104ddb02be23e7b4eda0765da875574c1b6b3483c140023"}
Sep 15 16:27:18.412951 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:18.412919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" event={"ID":"b84e787d-7468-4967-963a-bdf15591987a","Type":"ContainerStarted","Data":"e317c745adf420b9368f9a1eff8a74368db6bbeb1a6c33183e570ed12c42dcde"}
Sep 15 16:27:18.445127 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:18.445081    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h4rqc" podStartSLOduration=7.832836666 podStartE2EDuration="42.445065236s" totalImagesPullingTime="34.61222857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:18.445016283 +0000 UTC m=+42.780971216" watchObservedRunningTime="2026-09-15 16:27:18.445065236 +0000 UTC m=+42.781020163"
Sep 15 16:27:19.159846 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:19.159810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:19.159846 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:19.159830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:19.160058 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:19.159916    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:19.160058 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:19.159936    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:19.160058 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:19.159986    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:19.160153 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:19.160063    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:21.158888 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.158859    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:21.159275 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.158887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:21.159275 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:21.159128    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:21.159275 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.158887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:21.159275 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:21.159215    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:21.159518 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:21.159279    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:21.396111 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.396065    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5tx7l"]
Sep 15 16:27:21.400078 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.400052    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gt76w"]
Sep 15 16:27:21.408521 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.408459    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-db66d"]
Sep 15 16:27:21.419648 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.419627    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:21.419765 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.419660    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:21.419765 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:21.419675    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:21.419874 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:21.419791    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:21.419934 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:21.419913    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:21.420006 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:21.419986    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:23.158860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:23.158829    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:23.159435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:23.158826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:23.159435 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:23.158945    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:23.159435 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:23.159032    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:23.159435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:23.158826    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:23.159435 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:23.159136    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:25.159889 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.159702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:25.160323 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.159719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:25.160323 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:25.159957    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-db66d" podUID="584a2e9a-89de-477a-9450-abdbc0d5b622"
Sep 15 16:27:25.160323 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.159727    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:25.160323 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:25.160043    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gt76w" podUID="3209aee8-cc28-4cc4-a3f0-c6af9ede9a60"
Sep 15 16:27:25.160323 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:25.160148    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5tx7l" podUID="0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e"
Sep 15 16:27:25.405256 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.405227    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-221.ec2.internal" event="NodeReady"
Sep 15 16:27:25.405427 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.405358    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 16:27:25.464783 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.464748    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wsm9l"]
Sep 15 16:27:25.475944 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.475924    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8jm8c"]
Sep 15 16:27:25.476078 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.476061    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.478561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.478533    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 16:27:25.478684 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.478536    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m9qf9\""
Sep 15 16:27:25.478935 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.478800    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 16:27:25.489019 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.488999    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wsm9l"]
Sep 15 16:27:25.489124 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.489082    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.492184 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.492160    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 16:27:25.492184 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.492174    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 16:27:25.492691 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.492672    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 16:27:25.492824 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.492680    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 16:27:25.492824 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.492766    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zvqwt\""
Sep 15 16:27:25.495789 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.495773    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8jm8c"]
Sep 15 16:27:25.559001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.558927    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-prb6c"]
Sep 15 16:27:25.568222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.568203    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-prb6c"
Sep 15 16:27:25.571083 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.571065    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 16:27:25.571171 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.571094    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 16:27:25.571441 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.571424    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 16:27:25.571683 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.571667    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rwdks\""
Sep 15 16:27:25.577280 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.577260    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-prb6c"]
Sep 15 16:27:25.589823 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.589759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff240207-d755-42a9-9d1f-22f356486529-crio-socket\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.589904 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.589838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/326e7ec6-2ae7-4b30-b424-674d54fd4fce-metrics-tls\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.589961 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.589944    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf7p6\" (UniqueName: \"kubernetes.io/projected/ff240207-d755-42a9-9d1f-22f356486529-kube-api-access-xf7p6\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.590002 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.589971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff240207-d755-42a9-9d1f-22f356486529-data-volume\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.590054 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.590005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/326e7ec6-2ae7-4b30-b424-674d54fd4fce-config-volume\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.590105 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.590069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff240207-d755-42a9-9d1f-22f356486529-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.590105 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.590086    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vzmb\" (UniqueName: \"kubernetes.io/projected/326e7ec6-2ae7-4b30-b424-674d54fd4fce-kube-api-access-2vzmb\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.590197 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.590112    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff240207-d755-42a9-9d1f-22f356486529-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.590197 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.590146    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326e7ec6-2ae7-4b30-b424-674d54fd4fce-tmp-dir\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.691304 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51e32a97-593e-4eee-bca2-92e61c8dedf4-cert\") pod \"ingress-canary-prb6c\" (UID: \"51e32a97-593e-4eee-bca2-92e61c8dedf4\") " pod="openshift-ingress-canary/ingress-canary-prb6c"
Sep 15 16:27:25.691304 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691312    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/326e7ec6-2ae7-4b30-b424-674d54fd4fce-config-volume\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.691538 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691370    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff240207-d755-42a9-9d1f-22f356486529-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.691538 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vzmb\" (UniqueName: \"kubernetes.io/projected/326e7ec6-2ae7-4b30-b424-674d54fd4fce-kube-api-access-2vzmb\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.691538 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691440    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff240207-d755-42a9-9d1f-22f356486529-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.691538 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326e7ec6-2ae7-4b30-b424-674d54fd4fce-tmp-dir\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.691538 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff240207-d755-42a9-9d1f-22f356486529-crio-socket\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.691538 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691511    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/326e7ec6-2ae7-4b30-b424-674d54fd4fce-metrics-tls\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.691811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691668    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ff240207-d755-42a9-9d1f-22f356486529-crio-socket\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.691811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf7p6\" (UniqueName: \"kubernetes.io/projected/ff240207-d755-42a9-9d1f-22f356486529-kube-api-access-xf7p6\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.691811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff240207-d755-42a9-9d1f-22f356486529-data-volume\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.691811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691760    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mtxd\" (UniqueName: \"kubernetes.io/projected/51e32a97-593e-4eee-bca2-92e61c8dedf4-kube-api-access-6mtxd\") pod \"ingress-canary-prb6c\" (UID: \"51e32a97-593e-4eee-bca2-92e61c8dedf4\") " pod="openshift-ingress-canary/ingress-canary-prb6c"
Sep 15 16:27:25.691994 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.691936    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/326e7ec6-2ae7-4b30-b424-674d54fd4fce-tmp-dir\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.692109 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.692085    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ff240207-d755-42a9-9d1f-22f356486529-data-volume\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.692170 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.692138    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/326e7ec6-2ae7-4b30-b424-674d54fd4fce-config-volume\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.692728 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.692704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ff240207-d755-42a9-9d1f-22f356486529-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.695732 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.695715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/326e7ec6-2ae7-4b30-b424-674d54fd4fce-metrics-tls\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.699185 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.699165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vzmb\" (UniqueName: \"kubernetes.io/projected/326e7ec6-2ae7-4b30-b424-674d54fd4fce-kube-api-access-2vzmb\") pod \"dns-default-wsm9l\" (UID: \"326e7ec6-2ae7-4b30-b424-674d54fd4fce\") " pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.705382 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.705359    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ff240207-d755-42a9-9d1f-22f356486529-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.707073 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.707058    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf7p6\" (UniqueName: \"kubernetes.io/projected/ff240207-d755-42a9-9d1f-22f356486529-kube-api-access-xf7p6\") pod \"insights-runtime-extractor-8jm8c\" (UID: \"ff240207-d755-42a9-9d1f-22f356486529\") " pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.787248 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.787212    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:25.792530 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.792508    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mtxd\" (UniqueName: \"kubernetes.io/projected/51e32a97-593e-4eee-bca2-92e61c8dedf4-kube-api-access-6mtxd\") pod \"ingress-canary-prb6c\" (UID: \"51e32a97-593e-4eee-bca2-92e61c8dedf4\") " pod="openshift-ingress-canary/ingress-canary-prb6c"
Sep 15 16:27:25.792597 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.792537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51e32a97-593e-4eee-bca2-92e61c8dedf4-cert\") pod \"ingress-canary-prb6c\" (UID: \"51e32a97-593e-4eee-bca2-92e61c8dedf4\") " pod="openshift-ingress-canary/ingress-canary-prb6c"
Sep 15 16:27:25.794911 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.794892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/51e32a97-593e-4eee-bca2-92e61c8dedf4-cert\") pod \"ingress-canary-prb6c\" (UID: \"51e32a97-593e-4eee-bca2-92e61c8dedf4\") " pod="openshift-ingress-canary/ingress-canary-prb6c"
Sep 15 16:27:25.797904 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.797881    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8jm8c"
Sep 15 16:27:25.800699 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.800681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mtxd\" (UniqueName: \"kubernetes.io/projected/51e32a97-593e-4eee-bca2-92e61c8dedf4-kube-api-access-6mtxd\") pod \"ingress-canary-prb6c\" (UID: \"51e32a97-593e-4eee-bca2-92e61c8dedf4\") " pod="openshift-ingress-canary/ingress-canary-prb6c"
Sep 15 16:27:25.877288 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:25.877260    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-prb6c"
Sep 15 16:27:26.089947 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:26.089875    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wsm9l"]
Sep 15 16:27:26.092180 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:26.092154    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8jm8c"]
Sep 15 16:27:26.093969 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:26.093888    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326e7ec6_2ae7_4b30_b424_674d54fd4fce.slice/crio-3fe77305d03f8b227b8fdca68f8feab2ba22f7f35c5e9c1580a038660dbf8bc8 WatchSource:0}: Error finding container 3fe77305d03f8b227b8fdca68f8feab2ba22f7f35c5e9c1580a038660dbf8bc8: Status 404 returned error can't find the container with id 3fe77305d03f8b227b8fdca68f8feab2ba22f7f35c5e9c1580a038660dbf8bc8
Sep 15 16:27:26.095875 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:26.095848    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff240207_d755_42a9_9d1f_22f356486529.slice/crio-cc0343e1da0a90a5d74b3d63a1542ae2af125f57e1fe36826f46a64ac2bdb3c6 WatchSource:0}: Error finding container cc0343e1da0a90a5d74b3d63a1542ae2af125f57e1fe36826f46a64ac2bdb3c6: Status 404 returned error can't find the container with id cc0343e1da0a90a5d74b3d63a1542ae2af125f57e1fe36826f46a64ac2bdb3c6
Sep 15 16:27:26.154979 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:26.154950    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-prb6c"]
Sep 15 16:27:26.160093 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:26.160067    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e32a97_593e_4eee_bca2_92e61c8dedf4.slice/crio-c0ed352eaa745f2e09e3474fa025c22cfd7be703d56bdca316b3a4c94063a656 WatchSource:0}: Error finding container c0ed352eaa745f2e09e3474fa025c22cfd7be703d56bdca316b3a4c94063a656: Status 404 returned error can't find the container with id c0ed352eaa745f2e09e3474fa025c22cfd7be703d56bdca316b3a4c94063a656
Sep 15 16:27:26.431543 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:26.431253    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jm8c" event={"ID":"ff240207-d755-42a9-9d1f-22f356486529","Type":"ContainerStarted","Data":"61fba3825d35d390173487905475e58e9fca7ba91b8cbb5d7fdf4a9491044b73"}
Sep 15 16:27:26.431543 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:26.431520    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jm8c" event={"ID":"ff240207-d755-42a9-9d1f-22f356486529","Type":"ContainerStarted","Data":"cc0343e1da0a90a5d74b3d63a1542ae2af125f57e1fe36826f46a64ac2bdb3c6"}
Sep 15 16:27:26.432382 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:26.432361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-prb6c" event={"ID":"51e32a97-593e-4eee-bca2-92e61c8dedf4","Type":"ContainerStarted","Data":"c0ed352eaa745f2e09e3474fa025c22cfd7be703d56bdca316b3a4c94063a656"}
Sep 15 16:27:26.433387 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:26.433366    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wsm9l" event={"ID":"326e7ec6-2ae7-4b30-b424-674d54fd4fce","Type":"ContainerStarted","Data":"3fe77305d03f8b227b8fdca68f8feab2ba22f7f35c5e9c1580a038660dbf8bc8"}
Sep 15 16:27:27.159083 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.159029    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:27.159083 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.159146    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:27.159368 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.159182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:27.162359 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.162341    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 16:27:27.162728 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.162364    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 16:27:27.163312 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.163130    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v6gf7\""
Sep 15 16:27:27.163312 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.163214    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 16:27:27.165110 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.165090    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 16:27:27.166371 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.166182    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zqb5b\""
Sep 15 16:27:27.438591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:27.438507    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jm8c" event={"ID":"ff240207-d755-42a9-9d1f-22f356486529","Type":"ContainerStarted","Data":"4b18301e942548c90a7de820f0ffbfc9961c56e26d6e53472de49886fbeebdad"}
Sep 15 16:27:29.445148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:29.445077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-prb6c" event={"ID":"51e32a97-593e-4eee-bca2-92e61c8dedf4","Type":"ContainerStarted","Data":"680fe05d89791600df41a5c1b82561746ad544f23438bf2984fd207756cb0ed5"}
Sep 15 16:27:29.447561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:29.447535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wsm9l" event={"ID":"326e7ec6-2ae7-4b30-b424-674d54fd4fce","Type":"ContainerStarted","Data":"90de88255a5e8f55f051f752db1f5f4e7ca78db43911bb39fd491d3b86166d7c"}
Sep 15 16:27:29.447682 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:29.447579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wsm9l" event={"ID":"326e7ec6-2ae7-4b30-b424-674d54fd4fce","Type":"ContainerStarted","Data":"8efd88009c604079ed6372e7e77c93d7d027182860f5f58c5fffebed57e24f22"}
Sep 15 16:27:29.447682 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:29.447610    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:29.465307 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:29.465254    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-prb6c" podStartSLOduration=1.748829462 podStartE2EDuration="4.465235676s" totalImagesPullingTime="2.716406214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:29.465079674 +0000 UTC m=+53.801034606" watchObservedRunningTime="2026-09-15 16:27:29.465235676 +0000 UTC m=+53.801190601"
Sep 15 16:27:29.492579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:29.492532    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wsm9l" podStartSLOduration=1.7097659269999999 podStartE2EDuration="4.492513015s" totalImagesPullingTime="2.782747088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:29.492469 +0000 UTC m=+53.828423931" watchObservedRunningTime="2026-09-15 16:27:29.492513015 +0000 UTC m=+53.828467945"
Sep 15 16:27:30.453055 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:30.452822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8jm8c" event={"ID":"ff240207-d755-42a9-9d1f-22f356486529","Type":"ContainerStarted","Data":"3e22695bbf1ee7ebd5406fe45453cfeeee13db2d1059b92cca9537cc9fec8cde"}
Sep 15 16:27:30.477296 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:30.477250    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8jm8c" podStartSLOduration=1.8953736110000001 podStartE2EDuration="5.477235496s" totalImagesPullingTime="3.581861885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:30.476431916 +0000 UTC m=+54.812386846" watchObservedRunningTime="2026-09-15 16:27:30.477235496 +0000 UTC m=+54.813190426"
Sep 15 16:27:37.386883 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:37.386857    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcbrc"
Sep 15 16:27:39.455194 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:39.455157    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wsm9l"
Sep 15 16:27:40.901639 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:40.901595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:40.901639 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:40.901640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:40.905852 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:40.905832    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 16:27:40.906679 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:40.906665    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 16:27:40.916201 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:40.916180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e-original-pull-secret\") pod \"global-pull-secret-syncer-5tx7l\" (UID: \"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e\") " pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:40.916312 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:40.916226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3209aee8-cc28-4cc4-a3f0-c6af9ede9a60-metrics-certs\") pod \"network-metrics-daemon-gt76w\" (UID: \"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60\") " pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:41.002844 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.002815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:41.005503 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.005485    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 16:27:41.012063 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.012047    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5tx7l"
Sep 15 16:27:41.015544 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.015528    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 16:27:41.020017 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.019997    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zqb5b\""
Sep 15 16:27:41.026801 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.026780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bgb\" (UniqueName: \"kubernetes.io/projected/584a2e9a-89de-477a-9450-abdbc0d5b622-kube-api-access-v9bgb\") pod \"network-check-target-db66d\" (UID: \"584a2e9a-89de-477a-9450-abdbc0d5b622\") " pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:41.028355 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.028334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gt76w"
Sep 15 16:27:41.295901 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.295845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5tx7l"]
Sep 15 16:27:41.299182 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:41.299151    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1e7e97_0a0e_400a_ad6c_99bbd53dca7e.slice/crio-c267bf603203bb913cb1e147bfb1d4656c0da36dad46e09910fb64d5706dc8ce WatchSource:0}: Error finding container c267bf603203bb913cb1e147bfb1d4656c0da36dad46e09910fb64d5706dc8ce: Status 404 returned error can't find the container with id c267bf603203bb913cb1e147bfb1d4656c0da36dad46e09910fb64d5706dc8ce
Sep 15 16:27:41.301466 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.301433    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v6gf7\""
Sep 15 16:27:41.304522 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.304502    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:41.318749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.318726    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gt76w"]
Sep 15 16:27:41.323069 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:41.323045    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3209aee8_cc28_4cc4_a3f0_c6af9ede9a60.slice/crio-d5e0a740ab72e2fc2e4137f7b8be7c8c1c8ed2ccbdc0077470ef58e8e9cc85b2 WatchSource:0}: Error finding container d5e0a740ab72e2fc2e4137f7b8be7c8c1c8ed2ccbdc0077470ef58e8e9cc85b2: Status 404 returned error can't find the container with id d5e0a740ab72e2fc2e4137f7b8be7c8c1c8ed2ccbdc0077470ef58e8e9cc85b2
Sep 15 16:27:41.484815 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.484778    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5tx7l" event={"ID":"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e","Type":"ContainerStarted","Data":"c267bf603203bb913cb1e147bfb1d4656c0da36dad46e09910fb64d5706dc8ce"}
Sep 15 16:27:41.485889 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.485862    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gt76w" event={"ID":"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60","Type":"ContainerStarted","Data":"d5e0a740ab72e2fc2e4137f7b8be7c8c1c8ed2ccbdc0077470ef58e8e9cc85b2"}
Sep 15 16:27:41.583044 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:41.583020    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-db66d"]
Sep 15 16:27:41.585327 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:41.585301    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584a2e9a_89de_477a_9450_abdbc0d5b622.slice/crio-7b0f13fe39f3f76d9288fec2c7e1e828164f90c4b379a50d9efec8adce7d9751 WatchSource:0}: Error finding container 7b0f13fe39f3f76d9288fec2c7e1e828164f90c4b379a50d9efec8adce7d9751: Status 404 returned error can't find the container with id 7b0f13fe39f3f76d9288fec2c7e1e828164f90c4b379a50d9efec8adce7d9751
Sep 15 16:27:42.491406 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:42.491370    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-db66d" event={"ID":"584a2e9a-89de-477a-9450-abdbc0d5b622","Type":"ContainerStarted","Data":"7b0f13fe39f3f76d9288fec2c7e1e828164f90c4b379a50d9efec8adce7d9751"}
Sep 15 16:27:43.496947 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.496902    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gt76w" event={"ID":"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60","Type":"ContainerStarted","Data":"b49f976178c5fdff10b4d037128b723ced34ea60fd187a03fae74d8487e67dd5"}
Sep 15 16:27:43.497450 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.496976    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gt76w" event={"ID":"3209aee8-cc28-4cc4-a3f0-c6af9ede9a60","Type":"ContainerStarted","Data":"0eb07420a429d012c18ce2b90ae63049e51daf82007831335a88c3ced437d16f"}
Sep 15 16:27:43.517616 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.516527    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gt76w" podStartSLOduration=66.096335185 podStartE2EDuration="1m7.516506969s" totalImagesPullingTime="1.420171784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:43.516263404 +0000 UTC m=+67.852218327" watchObservedRunningTime="2026-09-15 16:27:43.516506969 +0000 UTC m=+67.852461903"
Sep 15 16:27:43.745107 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.745064    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"]
Sep 15 16:27:43.748233 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.748170    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"
Sep 15 16:27:43.751206 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.751183    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 16:27:43.751348 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.751185    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jtnld\""
Sep 15 16:27:43.770780 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.769532    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"]
Sep 15 16:27:43.922151 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.922115    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e2982d1-5e96-47f9-8941-dbf39b319934-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qx2l4\" (UID: \"9e2982d1-5e96-47f9-8941-dbf39b319934\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"
Sep 15 16:27:43.972720 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.972681    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-vnlcg"]
Sep 15 16:27:43.975815 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.975793    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-vnlcg"
Sep 15 16:27:43.979644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.979622    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 16:27:43.979753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.979691    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 16:27:43.980096 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.980083    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-4ht9r\""
Sep 15 16:27:43.991215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:43.991147    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-vnlcg"]
Sep 15 16:27:44.023440 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:44.023388    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e2982d1-5e96-47f9-8941-dbf39b319934-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qx2l4\" (UID: \"9e2982d1-5e96-47f9-8941-dbf39b319934\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"
Sep 15 16:27:44.023608 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:44.023529    2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 15 16:27:44.023652 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:44.023624    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e2982d1-5e96-47f9-8941-dbf39b319934-tls-certificates podName:9e2982d1-5e96-47f9-8941-dbf39b319934 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:44.523587698 +0000 UTC m=+68.859542609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9e2982d1-5e96-47f9-8941-dbf39b319934-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-qx2l4" (UID: "9e2982d1-5e96-47f9-8941-dbf39b319934") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 15 16:27:44.124031 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:44.123994    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8g95\" (UniqueName: \"kubernetes.io/projected/ad3e84de-141d-4b70-aeb4-f6f5663dae39-kube-api-access-p8g95\") pod \"downloads-76f99584f4-vnlcg\" (UID: \"ad3e84de-141d-4b70-aeb4-f6f5663dae39\") " pod="openshift-console/downloads-76f99584f4-vnlcg"
Sep 15 16:27:44.225279 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:44.225243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8g95\" (UniqueName: \"kubernetes.io/projected/ad3e84de-141d-4b70-aeb4-f6f5663dae39-kube-api-access-p8g95\") pod \"downloads-76f99584f4-vnlcg\" (UID: \"ad3e84de-141d-4b70-aeb4-f6f5663dae39\") " pod="openshift-console/downloads-76f99584f4-vnlcg"
Sep 15 16:27:44.236973 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:44.236944    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8g95\" (UniqueName: \"kubernetes.io/projected/ad3e84de-141d-4b70-aeb4-f6f5663dae39-kube-api-access-p8g95\") pod \"downloads-76f99584f4-vnlcg\" (UID: \"ad3e84de-141d-4b70-aeb4-f6f5663dae39\") " pod="openshift-console/downloads-76f99584f4-vnlcg"
Sep 15 16:27:44.287300 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:44.287207    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-vnlcg"
Sep 15 16:27:44.528271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:44.528237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e2982d1-5e96-47f9-8941-dbf39b319934-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qx2l4\" (UID: \"9e2982d1-5e96-47f9-8941-dbf39b319934\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"
Sep 15 16:27:44.531240 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:44.531215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9e2982d1-5e96-47f9-8941-dbf39b319934-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-qx2l4\" (UID: \"9e2982d1-5e96-47f9-8941-dbf39b319934\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"
Sep 15 16:27:44.659906 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:44.659649    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"
Sep 15 16:27:46.888271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:46.885776    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-vnlcg"]
Sep 15 16:27:46.890285 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:46.890207    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3e84de_141d_4b70_aeb4_f6f5663dae39.slice/crio-97dc213bffb8a0c5fa4442716113d25d7bbc7573309db940c8bf8d972b9c6e16 WatchSource:0}: Error finding container 97dc213bffb8a0c5fa4442716113d25d7bbc7573309db940c8bf8d972b9c6e16: Status 404 returned error can't find the container with id 97dc213bffb8a0c5fa4442716113d25d7bbc7573309db940c8bf8d972b9c6e16
Sep 15 16:27:46.891083 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:46.891045    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"]
Sep 15 16:27:46.896333 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:46.896304    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2982d1_5e96_47f9_8941_dbf39b319934.slice/crio-2b802ca5cdd3dcfbbcd94ed1756041aba42735b050d819cbd9a535f1a9752c1d WatchSource:0}: Error finding container 2b802ca5cdd3dcfbbcd94ed1756041aba42735b050d819cbd9a535f1a9752c1d: Status 404 returned error can't find the container with id 2b802ca5cdd3dcfbbcd94ed1756041aba42735b050d819cbd9a535f1a9752c1d
Sep 15 16:27:47.514963 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:47.514915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-vnlcg" event={"ID":"ad3e84de-141d-4b70-aeb4-f6f5663dae39","Type":"ContainerStarted","Data":"97dc213bffb8a0c5fa4442716113d25d7bbc7573309db940c8bf8d972b9c6e16"}
Sep 15 16:27:47.517115 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:47.517091    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5tx7l" event={"ID":"0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e","Type":"ContainerStarted","Data":"7cdc3757097aebea24f055986fc236fa7304e10a88661f0072c3302f9a246e93"}
Sep 15 16:27:47.519917 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:47.519893    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-db66d" event={"ID":"584a2e9a-89de-477a-9450-abdbc0d5b622","Type":"ContainerStarted","Data":"e59b146a131254b1461520bac748fbe2c1d32641c0c532e3790c9724ca41b7cf"}
Sep 15 16:27:47.520028 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:47.519952    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:27:47.522268 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:47.522243    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4" event={"ID":"9e2982d1-5e96-47f9-8941-dbf39b319934","Type":"ContainerStarted","Data":"2b802ca5cdd3dcfbbcd94ed1756041aba42735b050d819cbd9a535f1a9752c1d"}
Sep 15 16:27:47.533175 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:47.533135    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5tx7l" podStartSLOduration=66.209004427 podStartE2EDuration="1m11.533121621s" totalImagesPullingTime="5.324117194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:47.532898134 +0000 UTC m=+71.868853068" watchObservedRunningTime="2026-09-15 16:27:47.533121621 +0000 UTC m=+71.869076548"
Sep 15 16:27:47.548859 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:47.548810    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-db66d" podStartSLOduration=66.513209292 podStartE2EDuration="1m11.54879712s" totalImagesPullingTime="5.035587828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:47.54707818 +0000 UTC m=+71.883033112" watchObservedRunningTime="2026-09-15 16:27:47.54879712 +0000 UTC m=+71.884752052"
Sep 15 16:27:48.527551 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.527508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4" event={"ID":"9e2982d1-5e96-47f9-8941-dbf39b319934","Type":"ContainerStarted","Data":"53e786cbf92bb7e541f1620dc443e85f0b1a2fa0ffe4ff7c487631592e8d4b9f"}
Sep 15 16:27:48.528042 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.527749    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"
Sep 15 16:27:48.533356 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.533311    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4"
Sep 15 16:27:48.547314 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.547261    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-qx2l4" podStartSLOduration=4.469331806 podStartE2EDuration="5.54724127s" totalImagesPullingTime="1.077909464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:48.546279222 +0000 UTC m=+72.882234171" watchObservedRunningTime="2026-09-15 16:27:48.54724127 +0000 UTC m=+72.883196203"
Sep 15 16:27:48.806095 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.806016    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"]
Sep 15 16:27:48.809096 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.809068    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.812934 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.812915    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 16:27:48.812934 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.812928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 16:27:48.813728 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.813714    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vxsf5\""
Sep 15 16:27:48.813728 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.813721    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 16:27:48.813938 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.813923    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 16:27:48.814146 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.814130    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 16:27:48.818973 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.818952    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"]
Sep 15 16:27:48.856665 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.856631    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64da0150-d777-4d4f-93d9-d10d5eae5c06-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.856773 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.856688    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.856773 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.856763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qv5\" (UniqueName: \"kubernetes.io/projected/64da0150-d777-4d4f-93d9-d10d5eae5c06-kube-api-access-p5qv5\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.856863 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.856788    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.957508 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.957474    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qv5\" (UniqueName: \"kubernetes.io/projected/64da0150-d777-4d4f-93d9-d10d5eae5c06-kube-api-access-p5qv5\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.957508 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.957510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.957705 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.957542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64da0150-d777-4d4f-93d9-d10d5eae5c06-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.957705 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.957584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.957705 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:48.957662    2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 15 16:27:48.957810 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:48.957730    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-tls podName:64da0150-d777-4d4f-93d9-d10d5eae5c06 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:49.457709413 +0000 UTC m=+73.793664322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-srqlf" (UID: "64da0150-d777-4d4f-93d9-d10d5eae5c06") : secret "prometheus-operator-tls" not found
Sep 15 16:27:48.958367 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.958340    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/64da0150-d777-4d4f-93d9-d10d5eae5c06-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.960064 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.960042    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:48.968483 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:48.968460    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qv5\" (UniqueName: \"kubernetes.io/projected/64da0150-d777-4d4f-93d9-d10d5eae5c06-kube-api-access-p5qv5\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:49.460799 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:49.460764    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:49.463685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:49.463660    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/64da0150-d777-4d4f-93d9-d10d5eae5c06-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-srqlf\" (UID: \"64da0150-d777-4d4f-93d9-d10d5eae5c06\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:49.720983 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:49.720900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"
Sep 15 16:27:49.997492 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:49.997422    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-srqlf"]
Sep 15 16:27:50.000490 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:50.000460    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64da0150_d777_4d4f_93d9_d10d5eae5c06.slice/crio-32c38794bd4f1456a7462b8b64e0e7a122fbe0ed96575f22acf7adea56cf5233 WatchSource:0}: Error finding container 32c38794bd4f1456a7462b8b64e0e7a122fbe0ed96575f22acf7adea56cf5233: Status 404 returned error can't find the container with id 32c38794bd4f1456a7462b8b64e0e7a122fbe0ed96575f22acf7adea56cf5233
Sep 15 16:27:50.537265 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:50.537231    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf" event={"ID":"64da0150-d777-4d4f-93d9-d10d5eae5c06","Type":"ContainerStarted","Data":"32c38794bd4f1456a7462b8b64e0e7a122fbe0ed96575f22acf7adea56cf5233"}
Sep 15 16:27:51.543467 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:51.543436    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf" event={"ID":"64da0150-d777-4d4f-93d9-d10d5eae5c06","Type":"ContainerStarted","Data":"8f0bd7828a2ddc3c786b12c99b61f7f6c489a48fde1005cce41b0807e5582ada"}
Sep 15 16:27:52.549607 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:52.549566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf" event={"ID":"64da0150-d777-4d4f-93d9-d10d5eae5c06","Type":"ContainerStarted","Data":"c28dd4419fa37431fd3a92058e129d1be84d0cbef3ab6d0986e2d6053ca9cff1"}
Sep 15 16:27:52.569958 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:52.569910    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-srqlf" podStartSLOduration=3.174424953 podStartE2EDuration="4.56989335s" totalImagesPullingTime="1.395468397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:52.568755608 +0000 UTC m=+76.904710541" watchObservedRunningTime="2026-09-15 16:27:52.56989335 +0000 UTC m=+76.905848281"
Sep 15 16:27:54.013043 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.013006    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bd9fd95dc-c46tv"]
Sep 15 16:27:54.016164 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.016142    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.018901 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.018879    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 16:27:54.018901 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.018893    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 16:27:54.019807 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.019786    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 16:27:54.019980 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.019949    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4r7b8\""
Sep 15 16:27:54.020096 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.019983    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 16:27:54.020096 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.020020    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 16:27:54.026303 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.026275    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd9fd95dc-c46tv"]
Sep 15 16:27:54.099826 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.099794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-oauth-config\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.099993 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.099843    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-service-ca\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.099993 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.099932    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-serving-cert\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.100098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.099998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-console-config\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.100098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.100027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-oauth-serving-cert\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.100098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.100047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jbld\" (UniqueName: \"kubernetes.io/projected/f743da18-602f-485d-b71f-0042546134aa-kube-api-access-2jbld\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.201212 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.201181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-service-ca\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.201446 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.201229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-serving-cert\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.201446 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.201380    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-console-config\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.201578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.201447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-oauth-serving-cert\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.201578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.201476    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jbld\" (UniqueName: \"kubernetes.io/projected/f743da18-602f-485d-b71f-0042546134aa-kube-api-access-2jbld\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.201578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.201516    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-oauth-config\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.201940 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.201915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-service-ca\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.202516 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.202494    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-console-config\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.202516 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.202499    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-oauth-serving-cert\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.204213 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.204192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-oauth-config\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.204448 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.204424    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-serving-cert\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.211329 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.211296    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"]
Sep 15 16:27:54.214988 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.214968    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.219269 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.219251    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 16:27:54.219373 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.219355    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 16:27:54.219656 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.219642    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rl4xp\""
Sep 15 16:27:54.219881 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.219862    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 16:27:54.224604 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.224030    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-95pf2"]
Sep 15 16:27:54.227304 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.227285    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jbld\" (UniqueName: \"kubernetes.io/projected/f743da18-602f-485d-b71f-0042546134aa-kube-api-access-2jbld\") pod \"console-5bd9fd95dc-c46tv\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") " pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.227455 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.227437    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"]
Sep 15 16:27:54.227556 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.227544    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.230452 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.230432    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 16:27:54.230603 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.230583    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 16:27:54.230688 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.230662    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jpbg9\""
Sep 15 16:27:54.231578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.231379    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 16:27:54.301925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.301843    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.301925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.301874    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.301925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.301895    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.301925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.301912    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqzh4\" (UniqueName: \"kubernetes.io/projected/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-api-access-hqzh4\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.301976    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-accelerators-collector-config\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302008    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-tls\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302104    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-textfile\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-root\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302151    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcgg\" (UniqueName: \"kubernetes.io/projected/baa822fb-e94e-40f3-967e-6e4e04251bee-kube-api-access-wbcgg\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302168    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.302222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302227    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/baa822fb-e94e-40f3-967e-6e4e04251bee-metrics-client-ca\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.302645 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-wtmp\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.302645 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.302281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-sys\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.329662 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.329632    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-root\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406078    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcgg\" (UniqueName: \"kubernetes.io/projected/baa822fb-e94e-40f3-967e-6e4e04251bee-kube-api-access-wbcgg\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406109    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406139    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/baa822fb-e94e-40f3-967e-6e4e04251bee-metrics-client-ca\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406205    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-wtmp\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-sys\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406283    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqzh4\" (UniqueName: \"kubernetes.io/projected/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-api-access-hqzh4\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406386    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-accelerators-collector-config\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-tls\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-textfile\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.410685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406892    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-textfile\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.411867 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.406977    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-root\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.411867 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.408049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.411867 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.408883    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.411867 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.409362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/baa822fb-e94e-40f3-967e-6e4e04251bee-metrics-client-ca\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.411867 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.409543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-wtmp\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.411867 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.409595    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/baa822fb-e94e-40f3-967e-6e4e04251bee-sys\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.411867 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.410467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-accelerators-collector-config\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.415071 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.414320    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.419829 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.419497    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcgg\" (UniqueName: \"kubernetes.io/projected/baa822fb-e94e-40f3-967e-6e4e04251bee-kube-api-access-wbcgg\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.419829 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.419513    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.419829 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:54.419634    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 15 16:27:54.419829 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:27:54.419711    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-tls podName:baa822fb-e94e-40f3-967e-6e4e04251bee nodeName:}" failed. No retries permitted until 2026-09-15 16:27:54.919691523 +0000 UTC m=+79.255646451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-tls") pod "node-exporter-95pf2" (UID: "baa822fb-e94e-40f3-967e-6e4e04251bee") : secret "node-exporter-tls" not found
Sep 15 16:27:54.419829 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.419790    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.432793 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.432592    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqzh4\" (UniqueName: \"kubernetes.io/projected/cd210cff-3446-4ca9-98d0-b8b82b29ac4d-kube-api-access-hqzh4\") pod \"kube-state-metrics-59fbdd77d-p4kc4\" (UID: \"cd210cff-3446-4ca9-98d0-b8b82b29ac4d\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.434958 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.434932    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:54.529090 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.529052    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"
Sep 15 16:27:54.620720 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.620688    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bd9fd95dc-c46tv"]
Sep 15 16:27:54.624800 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:54.624774    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf743da18_602f_485d_b71f_0042546134aa.slice/crio-3c505e858e72c64939d1fa7faec07e8c55765cf36dc21453752f844b6497a7b3 WatchSource:0}: Error finding container 3c505e858e72c64939d1fa7faec07e8c55765cf36dc21453752f844b6497a7b3: Status 404 returned error can't find the container with id 3c505e858e72c64939d1fa7faec07e8c55765cf36dc21453752f844b6497a7b3
Sep 15 16:27:54.809047 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:54.808954    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4"]
Sep 15 16:27:54.812603 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:54.812576    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd210cff_3446_4ca9_98d0_b8b82b29ac4d.slice/crio-678882da5b28cd6777e1ea8995c9aed8028fa538fd90b6981a1c2ae6c466db4a WatchSource:0}: Error finding container 678882da5b28cd6777e1ea8995c9aed8028fa538fd90b6981a1c2ae6c466db4a: Status 404 returned error can't find the container with id 678882da5b28cd6777e1ea8995c9aed8028fa538fd90b6981a1c2ae6c466db4a
Sep 15 16:27:55.010447 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:55.010416    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-tls\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:55.012794 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:55.012772    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/baa822fb-e94e-40f3-967e-6e4e04251bee-node-exporter-tls\") pod \"node-exporter-95pf2\" (UID: \"baa822fb-e94e-40f3-967e-6e4e04251bee\") " pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:55.144021 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:55.143978    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-95pf2"
Sep 15 16:27:55.157108 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:27:55.157082    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa822fb_e94e_40f3_967e_6e4e04251bee.slice/crio-39d55380c80cf9fccde3df121f31304eb39377f298ad017431dce1270fb85faf WatchSource:0}: Error finding container 39d55380c80cf9fccde3df121f31304eb39377f298ad017431dce1270fb85faf: Status 404 returned error can't find the container with id 39d55380c80cf9fccde3df121f31304eb39377f298ad017431dce1270fb85faf
Sep 15 16:27:55.563179 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:55.563142    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95pf2" event={"ID":"baa822fb-e94e-40f3-967e-6e4e04251bee","Type":"ContainerStarted","Data":"39d55380c80cf9fccde3df121f31304eb39377f298ad017431dce1270fb85faf"}
Sep 15 16:27:55.565223 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:55.565184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd9fd95dc-c46tv" event={"ID":"f743da18-602f-485d-b71f-0042546134aa","Type":"ContainerStarted","Data":"3c505e858e72c64939d1fa7faec07e8c55765cf36dc21453752f844b6497a7b3"}
Sep 15 16:27:55.566765 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:55.566743    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4" event={"ID":"cd210cff-3446-4ca9-98d0-b8b82b29ac4d","Type":"ContainerStarted","Data":"678882da5b28cd6777e1ea8995c9aed8028fa538fd90b6981a1c2ae6c466db4a"}
Sep 15 16:27:58.538464 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.538310    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7fd897789f-hhsqg"]
Sep 15 16:27:58.542856 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.542830    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.546190 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.546167    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kmb8t\""
Sep 15 16:27:58.546317 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.546297    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 16:27:58.546447 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.546432    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 16:27:58.546549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.546534    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 16:27:58.546610 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.546167    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 16:27:58.546730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.546716    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c4c73956mg0b4\""
Sep 15 16:27:58.559084 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.559062    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fd897789f-hhsqg"]
Sep 15 16:27:58.639465 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.639422    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8424952e-1a56-4c1e-b194-6804e409d172-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.639641 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.639478    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-secret-metrics-server-client-certs\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.639641 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.639538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-secret-metrics-server-tls\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.639641 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.639586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8424952e-1a56-4c1e-b194-6804e409d172-metrics-server-audit-profiles\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.639641 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.639614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv92d\" (UniqueName: \"kubernetes.io/projected/8424952e-1a56-4c1e-b194-6804e409d172-kube-api-access-sv92d\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.639827 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.639681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8424952e-1a56-4c1e-b194-6804e409d172-audit-log\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.639827 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.639705    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-client-ca-bundle\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.740411 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.740353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8424952e-1a56-4c1e-b194-6804e409d172-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.740593 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.740423    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-secret-metrics-server-client-certs\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.740593 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.740456    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-secret-metrics-server-tls\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.740593 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.740490    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8424952e-1a56-4c1e-b194-6804e409d172-metrics-server-audit-profiles\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.740753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.740643    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv92d\" (UniqueName: \"kubernetes.io/projected/8424952e-1a56-4c1e-b194-6804e409d172-kube-api-access-sv92d\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.740753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.740708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8424952e-1a56-4c1e-b194-6804e409d172-audit-log\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.740753 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.740737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-client-ca-bundle\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.741240 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.741194    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8424952e-1a56-4c1e-b194-6804e409d172-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.741240 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.741227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8424952e-1a56-4c1e-b194-6804e409d172-audit-log\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.742048 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.742019    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8424952e-1a56-4c1e-b194-6804e409d172-metrics-server-audit-profiles\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.743988 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.743961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-secret-metrics-server-client-certs\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.744069 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.744001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-client-ca-bundle\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.744069 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.744063    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8424952e-1a56-4c1e-b194-6804e409d172-secret-metrics-server-tls\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.749302 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.749284    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv92d\" (UniqueName: \"kubernetes.io/projected/8424952e-1a56-4c1e-b194-6804e409d172-kube-api-access-sv92d\") pod \"metrics-server-7fd897789f-hhsqg\" (UID: \"8424952e-1a56-4c1e-b194-6804e409d172\") " pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.857014 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.856927    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:27:58.953578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.953550    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"]
Sep 15 16:27:58.956848 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.956824    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:27:58.959767 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.959719    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 16:27:58.959767 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.959732    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-x7v6c\""
Sep 15 16:27:58.965135 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:58.965110    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"]
Sep 15 16:27:59.044246 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:59.044212    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/058af8b2-822d-4644-b052-f62913a80db7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rjm4n\" (UID: \"058af8b2-822d-4644-b052-f62913a80db7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:27:59.044444 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:59.044326    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpvqx\" (UniqueName: \"kubernetes.io/projected/058af8b2-822d-4644-b052-f62913a80db7-kube-api-access-rpvqx\") pod \"monitoring-plugin-5d85778b9d-rjm4n\" (UID: \"058af8b2-822d-4644-b052-f62913a80db7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:27:59.145328 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:59.145244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/058af8b2-822d-4644-b052-f62913a80db7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rjm4n\" (UID: \"058af8b2-822d-4644-b052-f62913a80db7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:27:59.145517 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:59.145354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpvqx\" (UniqueName: \"kubernetes.io/projected/058af8b2-822d-4644-b052-f62913a80db7-kube-api-access-rpvqx\") pod \"monitoring-plugin-5d85778b9d-rjm4n\" (UID: \"058af8b2-822d-4644-b052-f62913a80db7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:27:59.148321 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:59.148290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/058af8b2-822d-4644-b052-f62913a80db7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-rjm4n\" (UID: \"058af8b2-822d-4644-b052-f62913a80db7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:27:59.154355 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:59.154326    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpvqx\" (UniqueName: \"kubernetes.io/projected/058af8b2-822d-4644-b052-f62913a80db7-kube-api-access-rpvqx\") pod \"monitoring-plugin-5d85778b9d-rjm4n\" (UID: \"058af8b2-822d-4644-b052-f62913a80db7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:27:59.271762 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:27:59.271725    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:28:00.400978 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.400942    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:28:00.405560 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.405533    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.408788 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.408747    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 16:28:00.408959 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.408879    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 16:28:00.409471 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.409006    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 16:28:00.409763 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.409695    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 16:28:00.409907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.409884    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 16:28:00.410074 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410045    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 16:28:00.410155 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410082    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 16:28:00.410239 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410215    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 16:28:00.410338 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410264    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 16:28:00.410568 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410529    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 16:28:00.410665 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410599    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 16:28:00.410665 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410636    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3fpinait9uu02\""
Sep 15 16:28:00.410781 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410727    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 16:28:00.410833 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410780    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rmqcx\""
Sep 15 16:28:00.410885 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.410861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 16:28:00.412751 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.412734    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 16:28:00.421443 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.421419    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:28:00.458326 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458592 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458711 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458711 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config-out\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458711 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458692    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-web-config\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458830 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458830 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458830 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458830 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.458830 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.458979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.459000    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.459038    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrf7\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-kube-api-access-tzrf7\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.459056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.459070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.459464 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.459094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.559687 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559650    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.559878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.559878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.559878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559769    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.559878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.559878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559826    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config-out\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.559878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-web-config\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.559878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559874    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559899    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559925    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559952    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.559976    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560143    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrf7\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-kube-api-access-tzrf7\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560197    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560843 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560368    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.560843 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.560701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.561201 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.561173    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.561892 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.561814    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.562597 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.562456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.563357 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.563328    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.564771 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.564725    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.564906 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.564878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.565098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.565074    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.566638 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.566598    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-web-config\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.566921 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.566899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.568028 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.567986    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.569081 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.569053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.569587 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.569548    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.569931 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.569893    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config-out\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.570478 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.570425    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.570993 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.570967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.571270 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.571246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrf7\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-kube-api-access-tzrf7\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.571621 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.571605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.572575 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.572551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:00.727543 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:00.727455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:05.610050 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:05.610023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4" event={"ID":"cd210cff-3446-4ca9-98d0-b8b82b29ac4d","Type":"ContainerStarted","Data":"795d6ee59bc5851b6e1c2536793f3ca0f5dc7953045555cff252bdfd61710b8d"}
Sep 15 16:28:05.743760 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:05.743681    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fd897789f-hhsqg"]
Sep 15 16:28:05.748767 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:05.748743    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"]
Sep 15 16:28:05.759767 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:28:05.759732    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8424952e_1a56_4c1e_b194_6804e409d172.slice/crio-a8a579bc5f0efdb2634e2dde7c31cb39d57bdd760f2659e2dafd292505345a38 WatchSource:0}: Error finding container a8a579bc5f0efdb2634e2dde7c31cb39d57bdd760f2659e2dafd292505345a38: Status 404 returned error can't find the container with id a8a579bc5f0efdb2634e2dde7c31cb39d57bdd760f2659e2dafd292505345a38
Sep 15 16:28:05.760656 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:28:05.760611    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058af8b2_822d_4644_b052_f62913a80db7.slice/crio-b5c4e8cb80cbad425b9556dc988886523b8daefa0da4604f157ac705fb35da46 WatchSource:0}: Error finding container b5c4e8cb80cbad425b9556dc988886523b8daefa0da4604f157ac705fb35da46: Status 404 returned error can't find the container with id b5c4e8cb80cbad425b9556dc988886523b8daefa0da4604f157ac705fb35da46
Sep 15 16:28:05.766539 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:05.766451    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:28:05.797526 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:28:05.797486    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be4c615_4f8a_4f51_a782_6af63e0e1dbb.slice/crio-cd6a3983149e6b813b935bda010badd624ac7ee0168814cc7353df196b9062b4 WatchSource:0}: Error finding container cd6a3983149e6b813b935bda010badd624ac7ee0168814cc7353df196b9062b4: Status 404 returned error can't find the container with id cd6a3983149e6b813b935bda010badd624ac7ee0168814cc7353df196b9062b4
Sep 15 16:28:06.619265 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.619221    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg" event={"ID":"8424952e-1a56-4c1e-b194-6804e409d172","Type":"ContainerStarted","Data":"a8a579bc5f0efdb2634e2dde7c31cb39d57bdd760f2659e2dafd292505345a38"}
Sep 15 16:28:06.622513 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.622485    2564 generic.go:356] "Generic (PLEG): container finished" podID="baa822fb-e94e-40f3-967e-6e4e04251bee" containerID="63c4009ad0f234c9627e49f7548748f7a21d21a71e797e98b3250598eb486b71" exitCode=0
Sep 15 16:28:06.622730 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.622696    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95pf2" event={"ID":"baa822fb-e94e-40f3-967e-6e4e04251bee","Type":"ContainerDied","Data":"63c4009ad0f234c9627e49f7548748f7a21d21a71e797e98b3250598eb486b71"}
Sep 15 16:28:06.625560 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.625175    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd9fd95dc-c46tv" event={"ID":"f743da18-602f-485d-b71f-0042546134aa","Type":"ContainerStarted","Data":"1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7"}
Sep 15 16:28:06.627139 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.627074    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n" event={"ID":"058af8b2-822d-4644-b052-f62913a80db7","Type":"ContainerStarted","Data":"b5c4e8cb80cbad425b9556dc988886523b8daefa0da4604f157ac705fb35da46"}
Sep 15 16:28:06.629457 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.629228    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-vnlcg" event={"ID":"ad3e84de-141d-4b70-aeb4-f6f5663dae39","Type":"ContainerStarted","Data":"9f8b3ddf2d9933b8e7d0517f79ce40ca6bbc0e30abce398a2a0e2ed04104993d"}
Sep 15 16:28:06.629941 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.629762    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-vnlcg"
Sep 15 16:28:06.633299 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.633273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4" event={"ID":"cd210cff-3446-4ca9-98d0-b8b82b29ac4d","Type":"ContainerStarted","Data":"545be71663e5e57d0d39b7ca931eaee956178e8afa0c4d2a90a0fe24e6bf46d1"}
Sep 15 16:28:06.633375 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.633327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4" event={"ID":"cd210cff-3446-4ca9-98d0-b8b82b29ac4d","Type":"ContainerStarted","Data":"0cfdd5d41cec69b23cb3e4d17abddaedfd039a804bf5108c822c16478a317627"}
Sep 15 16:28:06.635097 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.635068    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerStarted","Data":"cd6a3983149e6b813b935bda010badd624ac7ee0168814cc7353df196b9062b4"}
Sep 15 16:28:06.645148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.645118    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-vnlcg"
Sep 15 16:28:06.685040 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.684928    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bd9fd95dc-c46tv" podStartSLOduration=2.5297076609999998 podStartE2EDuration="13.684907892s" totalImagesPullingTime="11.155200231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:06.68385877 +0000 UTC m=+91.019813706" watchObservedRunningTime="2026-09-15 16:28:06.684907892 +0000 UTC m=+91.020862827"
Sep 15 16:28:06.711100 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.711050    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-vnlcg" podStartSLOduration=5.102120239 podStartE2EDuration="23.711031791s" totalImagesPullingTime="18.608911552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:06.708529766 +0000 UTC m=+91.044484697" watchObservedRunningTime="2026-09-15 16:28:06.711031791 +0000 UTC m=+91.046986721"
Sep 15 16:28:06.751936 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:06.750858    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-p4kc4" podStartSLOduration=2.132581329 podStartE2EDuration="12.750833596s" totalImagesPullingTime="10.618252267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:06.738582141 +0000 UTC m=+91.074537073" watchObservedRunningTime="2026-09-15 16:28:06.750833596 +0000 UTC m=+91.086788527"
Sep 15 16:28:07.644702 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:07.644616    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95pf2" event={"ID":"baa822fb-e94e-40f3-967e-6e4e04251bee","Type":"ContainerStarted","Data":"bf897521b9813690e71c9c886fbeedfe82c6de9ffc7fe52f9965c6c77bb8f099"}
Sep 15 16:28:09.655853 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.655816    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95pf2" event={"ID":"baa822fb-e94e-40f3-967e-6e4e04251bee","Type":"ContainerStarted","Data":"c6044ef2205958d70fb472235c0df5ea15f251f5ac8400002e13eb50714bd2cf"}
Sep 15 16:28:09.657822 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.657790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n" event={"ID":"058af8b2-822d-4644-b052-f62913a80db7","Type":"ContainerStarted","Data":"62db168635e6b6ea69ad9461ee99588c5ce9edaac74ed4ec3ad1a35885f2765d"}
Sep 15 16:28:09.657989 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.657963    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:28:09.659678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.659653    2564 generic.go:356] "Generic (PLEG): container finished" podID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerID="a9d9598329534dc0cb97c822306cfd7e590941c40b2a93b36b62cf43c84cd0a9" exitCode=0
Sep 15 16:28:09.659782 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.659747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerDied","Data":"a9d9598329534dc0cb97c822306cfd7e590941c40b2a93b36b62cf43c84cd0a9"}
Sep 15 16:28:09.663690 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.663067    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg" event={"ID":"8424952e-1a56-4c1e-b194-6804e409d172","Type":"ContainerStarted","Data":"a844867e3a76093ed28cf6d1630f480b694f82cffd3d392ab9be15b7e7e73b84"}
Sep 15 16:28:09.664734 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.664710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n"
Sep 15 16:28:09.681256 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.680670    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-95pf2" podStartSLOduration=5.27802058 podStartE2EDuration="15.680653444s" totalImagesPullingTime="10.282810593s" totalInitContainerRuntime="119.822271ms" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:09.678710227 +0000 UTC m=+94.014665194" watchObservedRunningTime="2026-09-15 16:28:09.680653444 +0000 UTC m=+94.016608376"
Sep 15 16:28:09.695561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:09.695514    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-rjm4n" podStartSLOduration=8.296309768 podStartE2EDuration="11.695498786s" totalImagesPullingTime="3.399189018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:09.693704148 +0000 UTC m=+94.029659079" watchObservedRunningTime="2026-09-15 16:28:09.695498786 +0000 UTC m=+94.031453725"
Sep 15 16:28:13.444338 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:13.444271    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg" podStartSLOduration=12.046074355 podStartE2EDuration="15.444246476s" totalImagesPullingTime="3.398172121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:09.751641213 +0000 UTC m=+94.087596144" watchObservedRunningTime="2026-09-15 16:28:13.444246476 +0000 UTC m=+97.780201407"
Sep 15 16:28:13.445258 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:13.445231    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bd9fd95dc-c46tv"]
Sep 15 16:28:14.330835 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:14.330792    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:28:14.694883 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:14.694849    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerStarted","Data":"a9157135c88b15ea202534cffcc139ff54877007ed0ecfc56af5059e5779ce1a"}
Sep 15 16:28:15.632500 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:15.632472    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-prb6c_51e32a97-593e-4eee-bca2-92e61c8dedf4/serve-healthcheck-canary/0.log"
Sep 15 16:28:15.702546 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:15.702509    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerStarted","Data":"1312d5f0f0ad598b68653c61c632dfe4be3095a1298ed5d7fdbffa5814d5fc52"}
Sep 15 16:28:17.713172 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:17.713085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerStarted","Data":"408e604873d94e35656fcd516bb8f34af981047c7a0c40afef0eedc195ae28d8"}
Sep 15 16:28:17.713172 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:17.713149    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerStarted","Data":"806eb49310ce4e1609683c962c464a1a0d980fe6299bb2d427496b513d209d8c"}
Sep 15 16:28:17.713617 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:17.713175    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerStarted","Data":"a9ba15c5a22477237142e196aef3bbc6b85f67cd73776f1f0f76432a36b02850"}
Sep 15 16:28:17.713617 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:17.713198    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerStarted","Data":"e214f8198f6f241ad96bed31b9ddbf1a7795b9d8751b01906ee8cf388491aacc"}
Sep 15 16:28:17.742307 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:17.742256    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.229101386 podStartE2EDuration="17.74223862s" totalImagesPullingTime="10.487255962s" totalInitContainerRuntime="25.881272ms" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:17.740632465 +0000 UTC m=+102.076587395" watchObservedRunningTime="2026-09-15 16:28:17.74223862 +0000 UTC m=+102.078193555"
Sep 15 16:28:18.529747 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:18.529717    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-db66d"
Sep 15 16:28:18.857832 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:18.857746    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:28:18.858190 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:18.857837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:28:20.728250 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:20.728221    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:28:38.472236 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.472172    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bd9fd95dc-c46tv" podUID="f743da18-602f-485d-b71f-0042546134aa" containerName="console" containerID="cri-o://1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7" gracePeriod=15
Sep 15 16:28:38.768080 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.768056    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:28:38.794793 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.794771    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-serving-cert\") pod \"f743da18-602f-485d-b71f-0042546134aa\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") "
Sep 15 16:28:38.794949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.794847    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jbld\" (UniqueName: \"kubernetes.io/projected/f743da18-602f-485d-b71f-0042546134aa-kube-api-access-2jbld\") pod \"f743da18-602f-485d-b71f-0042546134aa\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") "
Sep 15 16:28:38.794949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.794898    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-console-config\") pod \"f743da18-602f-485d-b71f-0042546134aa\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") "
Sep 15 16:28:38.794949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.794926    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-oauth-serving-cert\") pod \"f743da18-602f-485d-b71f-0042546134aa\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") "
Sep 15 16:28:38.795115 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.794951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-service-ca\") pod \"f743da18-602f-485d-b71f-0042546134aa\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") "
Sep 15 16:28:38.795115 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795075    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-oauth-config\") pod \"f743da18-602f-485d-b71f-0042546134aa\" (UID: \"f743da18-602f-485d-b71f-0042546134aa\") "
Sep 15 16:28:38.795313 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795284    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-console-config" (OuterVolumeSpecName: "console-config") pod "f743da18-602f-485d-b71f-0042546134aa" (UID: "f743da18-602f-485d-b71f-0042546134aa"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:28:38.795466 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795284    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f743da18-602f-485d-b71f-0042546134aa" (UID: "f743da18-602f-485d-b71f-0042546134aa"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:28:38.795550 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795525    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-service-ca" (OuterVolumeSpecName: "service-ca") pod "f743da18-602f-485d-b71f-0042546134aa" (UID: "f743da18-602f-485d-b71f-0042546134aa"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:28:38.795727 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795703    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bd9fd95dc-c46tv"
Sep 15 16:28:38.795802 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd9fd95dc-c46tv" event={"ID":"f743da18-602f-485d-b71f-0042546134aa","Type":"ContainerDied","Data":"1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7"}
Sep 15 16:28:38.795802 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795754    2564 generic.go:356] "Generic (PLEG): container finished" podID="f743da18-602f-485d-b71f-0042546134aa" containerID="1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7" exitCode=0
Sep 15 16:28:38.795897 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bd9fd95dc-c46tv" event={"ID":"f743da18-602f-485d-b71f-0042546134aa","Type":"ContainerDied","Data":"3c505e858e72c64939d1fa7faec07e8c55765cf36dc21453752f844b6497a7b3"}
Sep 15 16:28:38.795897 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.795855    2564 scope.go:117] "RemoveContainer" containerID="1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7"
Sep 15 16:28:38.797795 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.797766    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f743da18-602f-485d-b71f-0042546134aa" (UID: "f743da18-602f-485d-b71f-0042546134aa"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:28:38.797903 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.797856    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f743da18-602f-485d-b71f-0042546134aa" (UID: "f743da18-602f-485d-b71f-0042546134aa"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:28:38.797962 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.797911    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f743da18-602f-485d-b71f-0042546134aa-kube-api-access-2jbld" (OuterVolumeSpecName: "kube-api-access-2jbld") pod "f743da18-602f-485d-b71f-0042546134aa" (UID: "f743da18-602f-485d-b71f-0042546134aa"). InnerVolumeSpecName "kube-api-access-2jbld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:28:38.811016 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.810998    2564 scope.go:117] "RemoveContainer" containerID="1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7"
Sep 15 16:28:38.811297 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:28:38.811278    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7\": container with ID starting with 1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7 not found: ID does not exist" containerID="1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7"
Sep 15 16:28:38.811365 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.811309    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7"} err="failed to get container status \"1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7\": rpc error: code = NotFound desc = could not find container \"1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7\": container with ID starting with 1a83283a61cb7d528c212a426567de38dfde5a3e856b1a06a2ab0cb1ffc6e8a7 not found: ID does not exist"
Sep 15 16:28:38.862411 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.862371    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:28:38.866182 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.866160    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7fd897789f-hhsqg"
Sep 15 16:28:38.896227 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.896201    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-console-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:28:38.896227 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.896224    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-oauth-serving-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:28:38.896450 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.896236    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f743da18-602f-485d-b71f-0042546134aa-service-ca\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:28:38.896450 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.896250    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-oauth-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:28:38.896450 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.896262    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f743da18-602f-485d-b71f-0042546134aa-console-serving-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:28:38.896450 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:38.896275    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jbld\" (UniqueName: \"kubernetes.io/projected/f743da18-602f-485d-b71f-0042546134aa-kube-api-access-2jbld\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:28:39.116458 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:39.116429    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bd9fd95dc-c46tv"]
Sep 15 16:28:39.120207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:39.120183    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bd9fd95dc-c46tv"]
Sep 15 16:28:40.164124 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:28:40.164088    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f743da18-602f-485d-b71f-0042546134aa" path="/var/lib/kubelet/pods/f743da18-602f-485d-b71f-0042546134aa/volumes"
Sep 15 16:29:00.728209 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:00.728173    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:00.748497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:00.748471    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:00.889674 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:00.889645    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:18.769490 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.769457    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:18.770012 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.769797    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="prometheus" containerID="cri-o://a9157135c88b15ea202534cffcc139ff54877007ed0ecfc56af5059e5779ce1a" gracePeriod=600
Sep 15 16:29:18.770012 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.769857    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://408e604873d94e35656fcd516bb8f34af981047c7a0c40afef0eedc195ae28d8" gracePeriod=600
Sep 15 16:29:18.770012 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.769874    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy-web" containerID="cri-o://a9ba15c5a22477237142e196aef3bbc6b85f67cd73776f1f0f76432a36b02850" gracePeriod=600
Sep 15 16:29:18.770012 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.769887    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="config-reloader" containerID="cri-o://1312d5f0f0ad598b68653c61c632dfe4be3095a1298ed5d7fdbffa5814d5fc52" gracePeriod=600
Sep 15 16:29:18.770012 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.769893    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="thanos-sidecar" containerID="cri-o://e214f8198f6f241ad96bed31b9ddbf1a7795b9d8751b01906ee8cf388491aacc" gracePeriod=600
Sep 15 16:29:18.770012 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.769822    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy" containerID="cri-o://806eb49310ce4e1609683c962c464a1a0d980fe6299bb2d427496b513d209d8c" gracePeriod=600
Sep 15 16:29:18.944855 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.944825    2564 generic.go:356] "Generic (PLEG): container finished" podID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerID="408e604873d94e35656fcd516bb8f34af981047c7a0c40afef0eedc195ae28d8" exitCode=0
Sep 15 16:29:18.944855 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.944853    2564 generic.go:356] "Generic (PLEG): container finished" podID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerID="806eb49310ce4e1609683c962c464a1a0d980fe6299bb2d427496b513d209d8c" exitCode=0
Sep 15 16:29:18.944855 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.944861    2564 generic.go:356] "Generic (PLEG): container finished" podID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerID="a9ba15c5a22477237142e196aef3bbc6b85f67cd73776f1f0f76432a36b02850" exitCode=0
Sep 15 16:29:18.945121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.944867    2564 generic.go:356] "Generic (PLEG): container finished" podID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerID="e214f8198f6f241ad96bed31b9ddbf1a7795b9d8751b01906ee8cf388491aacc" exitCode=0
Sep 15 16:29:18.945121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.944871    2564 generic.go:356] "Generic (PLEG): container finished" podID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerID="1312d5f0f0ad598b68653c61c632dfe4be3095a1298ed5d7fdbffa5814d5fc52" exitCode=0
Sep 15 16:29:18.945121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.944877    2564 generic.go:356] "Generic (PLEG): container finished" podID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerID="a9157135c88b15ea202534cffcc139ff54877007ed0ecfc56af5059e5779ce1a" exitCode=0
Sep 15 16:29:18.945121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.944902    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerDied","Data":"408e604873d94e35656fcd516bb8f34af981047c7a0c40afef0eedc195ae28d8"}
Sep 15 16:29:18.945121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.944990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerDied","Data":"806eb49310ce4e1609683c962c464a1a0d980fe6299bb2d427496b513d209d8c"}
Sep 15 16:29:18.945121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.945033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerDied","Data":"a9ba15c5a22477237142e196aef3bbc6b85f67cd73776f1f0f76432a36b02850"}
Sep 15 16:29:18.945121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.945078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerDied","Data":"e214f8198f6f241ad96bed31b9ddbf1a7795b9d8751b01906ee8cf388491aacc"}
Sep 15 16:29:18.945121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.945120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerDied","Data":"1312d5f0f0ad598b68653c61c632dfe4be3095a1298ed5d7fdbffa5814d5fc52"}
Sep 15 16:29:18.945384 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:18.945160    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerDied","Data":"a9157135c88b15ea202534cffcc139ff54877007ed0ecfc56af5059e5779ce1a"}
Sep 15 16:29:19.019281 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.019255    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:19.115903 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.115819    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config-out\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.115903 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.115856    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-kubelet-serving-ca-bundle\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.115903 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.115874    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-1\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.115903 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.115901    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-trusted-ca-bundle\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.115928    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.115970    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-grpc-tls\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116013    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-0\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116056    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-metrics-client-ca\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116089    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzrf7\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-kube-api-access-tzrf7\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116122    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116149    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-thanos-prometheus-http-client-file\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116179    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116191    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-tls-assets\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116216    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-web-config\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116243    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116239    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116274    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-2\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116301    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-metrics-client-certs\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116328    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-kube-rbac-proxy\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116344    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-serving-certs-ca-bundle\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116494    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-db\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.116733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116523    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-tls\") pod \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\" (UID: \"6be4c615-4f8a-4f51-a782-6af63e0e1dbb\") "
Sep 15 16:29:19.117191 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116917    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.117191 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116938    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.117191 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.116954    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.117534 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.117498    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:19.117776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.117748    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:19.118129 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.118106    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:19.118726 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.118698    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:19.119667 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.119637    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config-out" (OuterVolumeSpecName: "config-out") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:29:19.119793 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.119756    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-kube-api-access-tzrf7" (OuterVolumeSpecName: "kube-api-access-tzrf7") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "kube-api-access-tzrf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:29:19.119793 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.119755    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:29:19.120248 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.120224    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.121002 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.120976    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.121091 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.121051    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.121250 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.121221    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.121250 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.121237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:29:19.121668 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.121635    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.121840 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.121819    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config" (OuterVolumeSpecName: "config") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.121956 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.121938    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.122449 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.122431    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.130455 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.130435    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-web-config" (OuterVolumeSpecName: "web-config") pod "6be4c615-4f8a-4f51-a782-6af63e0e1dbb" (UID: "6be4c615-4f8a-4f51-a782-6af63e0e1dbb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:19.217281 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217246    2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-grpc-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217281 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217273    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217281 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217285    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-metrics-client-ca\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217296    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzrf7\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-kube-api-access-tzrf7\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217304    2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217313    2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217322    2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-tls-assets\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217330    2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-web-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217338    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217347    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217355    2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-metrics-client-certs\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217364    2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-kube-rbac-proxy\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217372    2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217380    2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-prometheus-k8s-db\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217388    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217416    2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-config-out\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.217527 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.217429    2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6be4c615-4f8a-4f51-a782-6af63e0e1dbb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:29:19.951586 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.951549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6be4c615-4f8a-4f51-a782-6af63e0e1dbb","Type":"ContainerDied","Data":"cd6a3983149e6b813b935bda010badd624ac7ee0168814cc7353df196b9062b4"}
Sep 15 16:29:19.952031 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.951603    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:19.952031 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.951637    2564 scope.go:117] "RemoveContainer" containerID="408e604873d94e35656fcd516bb8f34af981047c7a0c40afef0eedc195ae28d8"
Sep 15 16:29:19.960117 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.960101    2564 scope.go:117] "RemoveContainer" containerID="806eb49310ce4e1609683c962c464a1a0d980fe6299bb2d427496b513d209d8c"
Sep 15 16:29:19.966721 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.966705    2564 scope.go:117] "RemoveContainer" containerID="a9ba15c5a22477237142e196aef3bbc6b85f67cd73776f1f0f76432a36b02850"
Sep 15 16:29:19.973044 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.973030    2564 scope.go:117] "RemoveContainer" containerID="e214f8198f6f241ad96bed31b9ddbf1a7795b9d8751b01906ee8cf388491aacc"
Sep 15 16:29:19.975125 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.975103    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:19.979958 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.979943    2564 scope.go:117] "RemoveContainer" containerID="1312d5f0f0ad598b68653c61c632dfe4be3095a1298ed5d7fdbffa5814d5fc52"
Sep 15 16:29:19.982854 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.982835    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:19.987049 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.987032    2564 scope.go:117] "RemoveContainer" containerID="a9157135c88b15ea202534cffcc139ff54877007ed0ecfc56af5059e5779ce1a"
Sep 15 16:29:19.994186 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:19.994168    2564 scope.go:117] "RemoveContainer" containerID="a9d9598329534dc0cb97c822306cfd7e590941c40b2a93b36b62cf43c84cd0a9"
Sep 15 16:29:20.011993 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.011974    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:20.012264 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012249    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy"
Sep 15 16:29:20.012336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012269    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy"
Sep 15 16:29:20.012336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012282    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy-web"
Sep 15 16:29:20.012336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012291    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy-web"
Sep 15 16:29:20.012336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012303    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="prometheus"
Sep 15 16:29:20.012336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012311    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="prometheus"
Sep 15 16:29:20.012336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012324    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f743da18-602f-485d-b71f-0042546134aa" containerName="console"
Sep 15 16:29:20.012336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012332    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f743da18-602f-485d-b71f-0042546134aa" containerName="console"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012340    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="thanos-sidecar"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012348    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="thanos-sidecar"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012359    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy-thanos"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012367    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy-thanos"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012384    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="init-config-reloader"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012412    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="init-config-reloader"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012423    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="config-reloader"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012432    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="config-reloader"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012508    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="config-reloader"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012519    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f743da18-602f-485d-b71f-0042546134aa" containerName="console"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012529    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="prometheus"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012536    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="thanos-sidecar"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012544    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy-thanos"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012555    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy-web"
Sep 15 16:29:20.012749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.012566    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" containerName="kube-rbac-proxy"
Sep 15 16:29:20.016193 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.016175    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019003    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019140    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019234    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019248    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019278    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019301    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3fpinait9uu02\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019309    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019327    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019235    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rmqcx\""
Sep 15 16:29:20.019436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019441    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 16:29:20.020060 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.019475    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 16:29:20.020118 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.020102    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 16:29:20.026747 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.026724    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 16:29:20.027550 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.027527    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 16:29:20.035902 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.035877    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:20.124989 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.124907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.124989 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.124947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.124989 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.124966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.124989 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.124981    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125050    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125141    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125167    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125223    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125320    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcvm\" (UniqueName: \"kubernetes.io/projected/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-kube-api-access-8fcvm\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125344    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125367    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125409    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.125576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.125432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-config\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.164354 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.164325    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be4c615-4f8a-4f51-a782-6af63e0e1dbb" path="/var/lib/kubelet/pods/6be4c615-4f8a-4f51-a782-6af63e0e1dbb/volumes"
Sep 15 16:29:20.225906 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.225876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226062 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.225917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226062 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.225943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226062 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.225964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226062 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.225986    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226062 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226062 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226040    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcvm\" (UniqueName: \"kubernetes.io/projected/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-kube-api-access-8fcvm\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226097    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226134    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226181    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226212    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-config\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226271    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226338    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226363    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226420    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226493    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226520    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.226758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.226546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.227343 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.227313    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.227493 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.227466    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.227609 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.227585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.228095 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.228064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.228871 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.228788    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.229116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.229081    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.229925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.229320    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-web-config\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.229925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.229878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.230074 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.229937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-config-out\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.230572 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.230527    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.230672 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.230586    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.230992 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.230957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.231700 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.231610    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.231829 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.231812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.232266 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.232244    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.232498 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.232479    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.232977 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.232957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.233326 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.233305    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-config\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.235716 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.235701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcvm\" (UniqueName: \"kubernetes.io/projected/061d28c3-a733-4c26-8c2a-477b9e6ff0cd-kube-api-access-8fcvm\") pod \"prometheus-k8s-0\" (UID: \"061d28c3-a733-4c26-8c2a-477b9e6ff0cd\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.329570 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.329529    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:29:20.607645 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.607556    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 16:29:20.609714 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:29:20.609687    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061d28c3_a733_4c26_8c2a_477b9e6ff0cd.slice/crio-646e810560b550a8948a87e67ea10c6e08fc348925113ccd4c83bcfd44a5a2a2 WatchSource:0}: Error finding container 646e810560b550a8948a87e67ea10c6e08fc348925113ccd4c83bcfd44a5a2a2: Status 404 returned error can't find the container with id 646e810560b550a8948a87e67ea10c6e08fc348925113ccd4c83bcfd44a5a2a2
Sep 15 16:29:20.956835 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.956748    2564 generic.go:356] "Generic (PLEG): container finished" podID="061d28c3-a733-4c26-8c2a-477b9e6ff0cd" containerID="bb08846c2ae88a49128d5fbac24f507530c3b7f78fb1bdc559ad4633cb841e6d" exitCode=0
Sep 15 16:29:20.957224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.956849    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"061d28c3-a733-4c26-8c2a-477b9e6ff0cd","Type":"ContainerDied","Data":"bb08846c2ae88a49128d5fbac24f507530c3b7f78fb1bdc559ad4633cb841e6d"}
Sep 15 16:29:20.957224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:20.956931    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"061d28c3-a733-4c26-8c2a-477b9e6ff0cd","Type":"ContainerStarted","Data":"646e810560b550a8948a87e67ea10c6e08fc348925113ccd4c83bcfd44a5a2a2"}
Sep 15 16:29:21.964659 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:21.964624    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"061d28c3-a733-4c26-8c2a-477b9e6ff0cd","Type":"ContainerStarted","Data":"b711af1f2e33cffd40cede4bb12625303483ad4326cbbd519d79571c7801247d"}
Sep 15 16:29:21.965082 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:21.964711    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"061d28c3-a733-4c26-8c2a-477b9e6ff0cd","Type":"ContainerStarted","Data":"1aeae42652222d92b6d6ed22da830bcaf13ab64be71b2adc7d6e677d36c408cd"}
Sep 15 16:29:21.965082 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:21.964736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"061d28c3-a733-4c26-8c2a-477b9e6ff0cd","Type":"ContainerStarted","Data":"6641d9eefc9b14e7cedc3d83b43af59fb9cb47eb5059fe173139ed92a98e5110"}
Sep 15 16:29:21.965082 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:21.964760    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"061d28c3-a733-4c26-8c2a-477b9e6ff0cd","Type":"ContainerStarted","Data":"5a7049f251f147c1ecb7620d6097ff8765656c1f1619c62ccc04649c660bdb36"}
Sep 15 16:29:21.965082 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:21.964787    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"061d28c3-a733-4c26-8c2a-477b9e6ff0cd","Type":"ContainerStarted","Data":"9adc456a796801f46911c8e98d5360a321e6d11bf17c85f13904c74b16d33b39"}
Sep 15 16:29:21.965082 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:21.964811    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"061d28c3-a733-4c26-8c2a-477b9e6ff0cd","Type":"ContainerStarted","Data":"2ef6007f11b5c1678fd6f3b1840078eba1c4642f261ab47d67a5ed672f086e64"}
Sep 15 16:29:21.998894 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:21.998846    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.980488926 podStartE2EDuration="2.99882881s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.339884ms" isStatefulPod=false podCreationTimestamp="2026-09-15 16:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:29:21.996101932 +0000 UTC m=+166.332056876" watchObservedRunningTime="2026-09-15 16:29:21.99882881 +0000 UTC m=+166.334783740"
Sep 15 16:29:25.329675 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:29:25.329630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:30:20.330208 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:30:20.330164    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:30:20.345814 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:30:20.345786    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:30:21.188430 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:30:21.188385    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 16:31:36.054804 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:31:36.054778    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 16:35:31.528726 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.528650    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-grwnl"]
Sep 15 16:35:31.530672 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.530656    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:31.536941 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.536641    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 15 16:35:31.537026 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.536939    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-759q5\""
Sep 15 16:35:31.537541 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.537524    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 15 16:35:31.537722 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.537707    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 15 16:35:31.555990 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.554237    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-grwnl"]
Sep 15 16:35:31.666598 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.666560    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9efc5be1-3a83-4513-a2e5-9441a6e48a65-data\") pod \"seaweedfs-64568bcd49-grwnl\" (UID: \"9efc5be1-3a83-4513-a2e5-9441a6e48a65\") " pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:31.666795 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.666657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f797g\" (UniqueName: \"kubernetes.io/projected/9efc5be1-3a83-4513-a2e5-9441a6e48a65-kube-api-access-f797g\") pod \"seaweedfs-64568bcd49-grwnl\" (UID: \"9efc5be1-3a83-4513-a2e5-9441a6e48a65\") " pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:31.767512 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.767475    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9efc5be1-3a83-4513-a2e5-9441a6e48a65-data\") pod \"seaweedfs-64568bcd49-grwnl\" (UID: \"9efc5be1-3a83-4513-a2e5-9441a6e48a65\") " pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:31.767636 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.767536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f797g\" (UniqueName: \"kubernetes.io/projected/9efc5be1-3a83-4513-a2e5-9441a6e48a65-kube-api-access-f797g\") pod \"seaweedfs-64568bcd49-grwnl\" (UID: \"9efc5be1-3a83-4513-a2e5-9441a6e48a65\") " pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:31.767918 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.767895    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/9efc5be1-3a83-4513-a2e5-9441a6e48a65-data\") pod \"seaweedfs-64568bcd49-grwnl\" (UID: \"9efc5be1-3a83-4513-a2e5-9441a6e48a65\") " pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:31.776262 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.776237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f797g\" (UniqueName: \"kubernetes.io/projected/9efc5be1-3a83-4513-a2e5-9441a6e48a65-kube-api-access-f797g\") pod \"seaweedfs-64568bcd49-grwnl\" (UID: \"9efc5be1-3a83-4513-a2e5-9441a6e48a65\") " pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:31.839384 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:31.839299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:32.108479 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:32.108378    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-grwnl"]
Sep 15 16:35:32.111460 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:35:32.111431    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efc5be1_3a83_4513_a2e5_9441a6e48a65.slice/crio-ab34efe45384d2767f5c425dd5b6b4d88e9cb55253794315adcaa6abb03d0f7d WatchSource:0}: Error finding container ab34efe45384d2767f5c425dd5b6b4d88e9cb55253794315adcaa6abb03d0f7d: Status 404 returned error can't find the container with id ab34efe45384d2767f5c425dd5b6b4d88e9cb55253794315adcaa6abb03d0f7d
Sep 15 16:35:32.113033 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:32.113012    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:35:32.252959 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:32.252921    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-grwnl" event={"ID":"9efc5be1-3a83-4513-a2e5-9441a6e48a65","Type":"ContainerStarted","Data":"ab34efe45384d2767f5c425dd5b6b4d88e9cb55253794315adcaa6abb03d0f7d"}
Sep 15 16:35:36.267715 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:36.267673    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-grwnl" event={"ID":"9efc5be1-3a83-4513-a2e5-9441a6e48a65","Type":"ContainerStarted","Data":"a5909bdacf0416b21e2eac9b5a6e78d51d19184accd31a9c7be68e28edf47930"}
Sep 15 16:35:36.268067 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:36.267818    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:35:36.283209 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:36.283163    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-grwnl" podStartSLOduration=2.137538169 podStartE2EDuration="5.28314774s" totalImagesPullingTime="3.145609571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:35:36.282519169 +0000 UTC m=+540.618474105" watchObservedRunningTime="2026-09-15 16:35:36.28314774 +0000 UTC m=+540.619102671"
Sep 15 16:35:42.273622 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:35:42.273587    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-grwnl"
Sep 15 16:36:49.489331 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.489299    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-wfvf8"]
Sep 15 16:36:49.492686 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.492662    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:49.495367 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.495344    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 15 16:36:49.495367 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.495362    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-87szh\""
Sep 15 16:36:49.502299 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.502271    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wfvf8"]
Sep 15 16:36:49.504594 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.504569    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-xbhcn"]
Sep 15 16:36:49.508011 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.507988    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.511036 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.511014    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 15 16:36:49.511355 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.511339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-njgz9\""
Sep 15 16:36:49.511644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.511629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 15 16:36:49.517095 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.517071    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-xbhcn"]
Sep 15 16:36:49.553651 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.553613    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpnq\" (UniqueName: \"kubernetes.io/projected/97fef691-d4b9-42b9-87f0-46d131ee1fc3-kube-api-access-nqpnq\") pod \"model-serving-api-7b954ffc95-wfvf8\" (UID: \"97fef691-d4b9-42b9-87f0-46d131ee1fc3\") " pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:49.553651 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.553654    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fef691-d4b9-42b9-87f0-46d131ee1fc3-tls-certs\") pod \"model-serving-api-7b954ffc95-wfvf8\" (UID: \"97fef691-d4b9-42b9-87f0-46d131ee1fc3\") " pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:49.553860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.553683    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-metrics-certs\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.553860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.553743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srzb\" (UniqueName: \"kubernetes.io/projected/da96b0c7-4cee-4070-8524-09a675071919-kube-api-access-2srzb\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.553860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.553808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-cert\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.654775 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.654735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-cert\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.654954 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.654792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpnq\" (UniqueName: \"kubernetes.io/projected/97fef691-d4b9-42b9-87f0-46d131ee1fc3-kube-api-access-nqpnq\") pod \"model-serving-api-7b954ffc95-wfvf8\" (UID: \"97fef691-d4b9-42b9-87f0-46d131ee1fc3\") " pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:49.654954 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.654810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fef691-d4b9-42b9-87f0-46d131ee1fc3-tls-certs\") pod \"model-serving-api-7b954ffc95-wfvf8\" (UID: \"97fef691-d4b9-42b9-87f0-46d131ee1fc3\") " pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:49.654954 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.654829    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-metrics-certs\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.654954 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.654852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srzb\" (UniqueName: \"kubernetes.io/projected/da96b0c7-4cee-4070-8524-09a675071919-kube-api-access-2srzb\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.654954 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:36:49.654927    2564 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 15 16:36:49.655146 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:36:49.655002    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-metrics-certs podName:da96b0c7-4cee-4070-8524-09a675071919 nodeName:}" failed. No retries permitted until 2026-09-15 16:36:50.154979237 +0000 UTC m=+614.490934147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-metrics-certs") pod "odh-model-controller-7b966d7984-xbhcn" (UID: "da96b0c7-4cee-4070-8524-09a675071919") : secret "odh-model-controller-metrics-tls" not found
Sep 15 16:36:49.657555 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.657524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97fef691-d4b9-42b9-87f0-46d131ee1fc3-tls-certs\") pod \"model-serving-api-7b954ffc95-wfvf8\" (UID: \"97fef691-d4b9-42b9-87f0-46d131ee1fc3\") " pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:49.657684 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.657635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-cert\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.663602 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.663569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpnq\" (UniqueName: \"kubernetes.io/projected/97fef691-d4b9-42b9-87f0-46d131ee1fc3-kube-api-access-nqpnq\") pod \"model-serving-api-7b954ffc95-wfvf8\" (UID: \"97fef691-d4b9-42b9-87f0-46d131ee1fc3\") " pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:49.663733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.663689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srzb\" (UniqueName: \"kubernetes.io/projected/da96b0c7-4cee-4070-8524-09a675071919-kube-api-access-2srzb\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:49.804549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:49.804511    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:50.078234 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:50.078203    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-wfvf8"]
Sep 15 16:36:50.081205 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:36:50.081173    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fef691_d4b9_42b9_87f0_46d131ee1fc3.slice/crio-806cc6433f6f6ed530f0b071d833a73cf40b7ade02e5d4533e2e63902667e8bb WatchSource:0}: Error finding container 806cc6433f6f6ed530f0b071d833a73cf40b7ade02e5d4533e2e63902667e8bb: Status 404 returned error can't find the container with id 806cc6433f6f6ed530f0b071d833a73cf40b7ade02e5d4533e2e63902667e8bb
Sep 15 16:36:50.159382 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:50.159343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-metrics-certs\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:50.162051 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:50.162027    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da96b0c7-4cee-4070-8524-09a675071919-metrics-certs\") pod \"odh-model-controller-7b966d7984-xbhcn\" (UID: \"da96b0c7-4cee-4070-8524-09a675071919\") " pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:50.418937 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:50.418837    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:50.527826 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:50.527786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wfvf8" event={"ID":"97fef691-d4b9-42b9-87f0-46d131ee1fc3","Type":"ContainerStarted","Data":"806cc6433f6f6ed530f0b071d833a73cf40b7ade02e5d4533e2e63902667e8bb"}
Sep 15 16:36:50.694511 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:50.694387    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-xbhcn"]
Sep 15 16:36:50.699743 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:36:50.699692    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda96b0c7_4cee_4070_8524_09a675071919.slice/crio-c6975f3c9f883a83c02f6d96986e2699ec94fe5d6ae47405489f4beba9fae2e0 WatchSource:0}: Error finding container c6975f3c9f883a83c02f6d96986e2699ec94fe5d6ae47405489f4beba9fae2e0: Status 404 returned error can't find the container with id c6975f3c9f883a83c02f6d96986e2699ec94fe5d6ae47405489f4beba9fae2e0
Sep 15 16:36:51.535289 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:51.535201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-xbhcn" event={"ID":"da96b0c7-4cee-4070-8524-09a675071919","Type":"ContainerStarted","Data":"c6975f3c9f883a83c02f6d96986e2699ec94fe5d6ae47405489f4beba9fae2e0"}
Sep 15 16:36:54.550833 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:54.550792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-wfvf8" event={"ID":"97fef691-d4b9-42b9-87f0-46d131ee1fc3","Type":"ContainerStarted","Data":"eb6f64a007a3c37135dbaf26f1a619c47213283e20a1ae7452d4fed0ff939ab1"}
Sep 15 16:36:54.551231 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:54.550895    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:36:54.552352 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:54.552326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-xbhcn" event={"ID":"da96b0c7-4cee-4070-8524-09a675071919","Type":"ContainerStarted","Data":"9ff36ebe599c8b3555aa79dc9c1bcd31665a6d6192d9bcc1db4a312f31fc98b4"}
Sep 15 16:36:54.552475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:54.552433    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:36:54.567006 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:54.566948    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-wfvf8" podStartSLOduration=1.823485774 podStartE2EDuration="5.566930617s" totalImagesPullingTime="3.743444843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:36:54.56526655 +0000 UTC m=+618.901221481" watchObservedRunningTime="2026-09-15 16:36:54.566930617 +0000 UTC m=+618.902885548"
Sep 15 16:36:54.581626 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:36:54.581573    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-xbhcn" podStartSLOduration=2.414767189 podStartE2EDuration="5.581557736s" totalImagesPullingTime="3.166790547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:36:54.580597831 +0000 UTC m=+618.916552787" watchObservedRunningTime="2026-09-15 16:36:54.581557736 +0000 UTC m=+618.917512668"
Sep 15 16:37:05.559216 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:05.559136    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-xbhcn"
Sep 15 16:37:05.561303 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:05.561286    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-wfvf8"
Sep 15 16:37:06.353504 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.353470    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-5ztmz"]
Sep 15 16:37:06.360083 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.360064    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5ztmz"
Sep 15 16:37:06.362595 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.362568    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 15 16:37:06.367967 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.367947    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5ztmz"]
Sep 15 16:37:06.392322 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.392295    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db490de3-5409-4751-a244-8e0e00862792-model-cache\") pod \"s3-init-5ztmz\" (UID: \"db490de3-5409-4751-a244-8e0e00862792\") " pod="kserve/s3-init-5ztmz"
Sep 15 16:37:06.392468 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.392341    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbwj\" (UniqueName: \"kubernetes.io/projected/db490de3-5409-4751-a244-8e0e00862792-kube-api-access-rtbwj\") pod \"s3-init-5ztmz\" (UID: \"db490de3-5409-4751-a244-8e0e00862792\") " pod="kserve/s3-init-5ztmz"
Sep 15 16:37:06.493698 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.493648    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db490de3-5409-4751-a244-8e0e00862792-model-cache\") pod \"s3-init-5ztmz\" (UID: \"db490de3-5409-4751-a244-8e0e00862792\") " pod="kserve/s3-init-5ztmz"
Sep 15 16:37:06.493860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.493716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbwj\" (UniqueName: \"kubernetes.io/projected/db490de3-5409-4751-a244-8e0e00862792-kube-api-access-rtbwj\") pod \"s3-init-5ztmz\" (UID: \"db490de3-5409-4751-a244-8e0e00862792\") " pod="kserve/s3-init-5ztmz"
Sep 15 16:37:06.494021 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.494001    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db490de3-5409-4751-a244-8e0e00862792-model-cache\") pod \"s3-init-5ztmz\" (UID: \"db490de3-5409-4751-a244-8e0e00862792\") " pod="kserve/s3-init-5ztmz"
Sep 15 16:37:06.501694 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.501666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbwj\" (UniqueName: \"kubernetes.io/projected/db490de3-5409-4751-a244-8e0e00862792-kube-api-access-rtbwj\") pod \"s3-init-5ztmz\" (UID: \"db490de3-5409-4751-a244-8e0e00862792\") " pod="kserve/s3-init-5ztmz"
Sep 15 16:37:06.680155 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.680077    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5ztmz"
Sep 15 16:37:06.949454 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:06.949347    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5ztmz"]
Sep 15 16:37:06.953636 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:37:06.953607    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb490de3_5409_4751_a244_8e0e00862792.slice/crio-32907a40c7a0592c18087b8a9ed693e8c49f76767d8fb7d2202029dfa99d1a0d WatchSource:0}: Error finding container 32907a40c7a0592c18087b8a9ed693e8c49f76767d8fb7d2202029dfa99d1a0d: Status 404 returned error can't find the container with id 32907a40c7a0592c18087b8a9ed693e8c49f76767d8fb7d2202029dfa99d1a0d
Sep 15 16:37:07.603657 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:07.603626    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5ztmz" event={"ID":"db490de3-5409-4751-a244-8e0e00862792","Type":"ContainerStarted","Data":"32907a40c7a0592c18087b8a9ed693e8c49f76767d8fb7d2202029dfa99d1a0d"}
Sep 15 16:37:11.622026 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:11.621990    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5ztmz" event={"ID":"db490de3-5409-4751-a244-8e0e00862792","Type":"ContainerStarted","Data":"3e69fd6ee0f76b065bace3785efff845d0b410a8c60cd11bad81562a46bd545e"}
Sep 15 16:37:13.631190 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:13.631158    2564 generic.go:356] "Generic (PLEG): container finished" podID="db490de3-5409-4751-a244-8e0e00862792" containerID="3e69fd6ee0f76b065bace3785efff845d0b410a8c60cd11bad81562a46bd545e" exitCode=0
Sep 15 16:37:13.631562 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:13.631236    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5ztmz" event={"ID":"db490de3-5409-4751-a244-8e0e00862792","Type":"ContainerDied","Data":"3e69fd6ee0f76b065bace3785efff845d0b410a8c60cd11bad81562a46bd545e"}
Sep 15 16:37:18.663783 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:18.663744    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5ztmz" event={"ID":"db490de3-5409-4751-a244-8e0e00862792","Type":"ContainerStarted","Data":"eb0c14f13a8fd1469c1081d40c22005b7bbe4fb5c1b393cf665b0dde7a1782d2"}
Sep 15 16:37:18.678364 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:37:18.678315    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-5ztmz" podStartSLOduration=2.509116438 podStartE2EDuration="12.678300895s" totalImagesPullingTime="8.308161392s" totalInitContainerRuntime="1.861023065s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:37:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:37:18.677295358 +0000 UTC m=+643.013250362" watchObservedRunningTime="2026-09-15 16:37:18.678300895 +0000 UTC m=+643.014255825"
Sep 15 16:38:23.905164 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:23.905134    2564 generic.go:356] "Generic (PLEG): container finished" podID="db490de3-5409-4751-a244-8e0e00862792" containerID="eb0c14f13a8fd1469c1081d40c22005b7bbe4fb5c1b393cf665b0dde7a1782d2" exitCode=0
Sep 15 16:38:23.905592 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:23.905172    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5ztmz" event={"ID":"db490de3-5409-4751-a244-8e0e00862792","Type":"ContainerDied","Data":"eb0c14f13a8fd1469c1081d40c22005b7bbe4fb5c1b393cf665b0dde7a1782d2"}
Sep 15 16:38:25.035472 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.035448    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5ztmz"
Sep 15 16:38:25.142616 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.142583    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db490de3-5409-4751-a244-8e0e00862792-model-cache\") pod \"db490de3-5409-4751-a244-8e0e00862792\" (UID: \"db490de3-5409-4751-a244-8e0e00862792\") "
Sep 15 16:38:25.142800 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.142623    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtbwj\" (UniqueName: \"kubernetes.io/projected/db490de3-5409-4751-a244-8e0e00862792-kube-api-access-rtbwj\") pod \"db490de3-5409-4751-a244-8e0e00862792\" (UID: \"db490de3-5409-4751-a244-8e0e00862792\") "
Sep 15 16:38:25.145099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.145061    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db490de3-5409-4751-a244-8e0e00862792-kube-api-access-rtbwj" (OuterVolumeSpecName: "kube-api-access-rtbwj") pod "db490de3-5409-4751-a244-8e0e00862792" (UID: "db490de3-5409-4751-a244-8e0e00862792"). InnerVolumeSpecName "kube-api-access-rtbwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:38:25.160930 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.160872    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db490de3-5409-4751-a244-8e0e00862792-model-cache" (OuterVolumeSpecName: "model-cache") pod "db490de3-5409-4751-a244-8e0e00862792" (UID: "db490de3-5409-4751-a244-8e0e00862792"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:38:25.244157 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.244129    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db490de3-5409-4751-a244-8e0e00862792-model-cache\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:38:25.244157 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.244152    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtbwj\" (UniqueName: \"kubernetes.io/projected/db490de3-5409-4751-a244-8e0e00862792-kube-api-access-rtbwj\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:38:25.913818 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.913793    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5ztmz"
Sep 15 16:38:25.913983 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.913790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5ztmz" event={"ID":"db490de3-5409-4751-a244-8e0e00862792","Type":"ContainerDied","Data":"32907a40c7a0592c18087b8a9ed693e8c49f76767d8fb7d2202029dfa99d1a0d"}
Sep 15 16:38:25.913983 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:25.913898    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32907a40c7a0592c18087b8a9ed693e8c49f76767d8fb7d2202029dfa99d1a0d"
Sep 15 16:38:38.187285 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.187253    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"]
Sep 15 16:38:38.187686 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.187583    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db490de3-5409-4751-a244-8e0e00862792" containerName="download-hf-model"
Sep 15 16:38:38.187686 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.187596    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db490de3-5409-4751-a244-8e0e00862792" containerName="download-hf-model"
Sep 15 16:38:38.187686 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.187610    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db490de3-5409-4751-a244-8e0e00862792" containerName="s3-init"
Sep 15 16:38:38.187686 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.187616    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="db490de3-5409-4751-a244-8e0e00862792" containerName="s3-init"
Sep 15 16:38:38.187686 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.187673    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="db490de3-5409-4751-a244-8e0e00862792" containerName="s3-init"
Sep 15 16:38:38.191038 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.191022    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.198117 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.198096    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-d4131-predictor-serving-cert\""
Sep 15 16:38:38.198215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.198201    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lqw4s\""
Sep 15 16:38:38.198731 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.198716    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-d4131-kube-rbac-proxy-sar-config\""
Sep 15 16:38:38.198903 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.198865    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:38:38.199020 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.199002    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:38:38.199076 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.199013    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:38:38.200778 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.200765    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:38:38.204829 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.204808    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"]
Sep 15 16:38:38.244445 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.244406    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjh6t\" (UniqueName: \"kubernetes.io/projected/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-api-access-tjh6t\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.244602 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.244452    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.244602 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.244478    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.244602 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.244507    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.244602 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.244527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.345735 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.345700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjh6t\" (UniqueName: \"kubernetes.io/projected/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-api-access-tjh6t\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.345735 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.345742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.345964 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.345763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.345964 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.345794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.345964 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.345821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.345964 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:38:38.345939    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-d4131-predictor-serving-cert" not found
Sep 15 16:38:38.346157 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:38:38.346041    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls podName:2e7ede59-a6f5-4e3b-872a-f0dbc27e623a nodeName:}" failed. No retries permitted until 2026-09-15 16:38:38.84601712 +0000 UTC m=+723.181972056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls") pod "isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" (UID: "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a") : secret "isvc-raw-sklearn-batcher-d4131-predictor-serving-cert" not found
Sep 15 16:38:38.346463 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.346433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.346591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.346495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.346846 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.346824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.354488 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.354458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjh6t\" (UniqueName: \"kubernetes.io/projected/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-api-access-tjh6t\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.377021 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.376992    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"]
Sep 15 16:38:38.380595 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.380576    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.383134 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.383115    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 15 16:38:38.383206 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.383138    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 15 16:38:38.392218 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.392197    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"]
Sep 15 16:38:38.411872 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.411842    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"]
Sep 15 16:38:38.415415 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.415372    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.417684 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.417664    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 15 16:38:38.425848 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.425829    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"]
Sep 15 16:38:38.446436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446355    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txbxc\" (UniqueName: \"kubernetes.io/projected/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-api-access-txbxc\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.446436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446384    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0628001b-71a5-4dce-bfca-3559f031c99f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.446610 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.446610 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446464    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.446610 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446483    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.446610 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.446610 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446610    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.446899 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xpgx\" (UniqueName: \"kubernetes.io/projected/0628001b-71a5-4dce-bfca-3559f031c99f-kube-api-access-7xpgx\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.446899 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.446899 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.446668    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.547812 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.547776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.548010 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.547830    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.548010 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.547862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.548010 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.547913    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.548010 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:38:38.547937    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 15 16:38:38.548010 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.547971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.548010 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xpgx\" (UniqueName: \"kubernetes.io/projected/0628001b-71a5-4dce-bfca-3559f031c99f-kube-api-access-7xpgx\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.548301 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:38:38.548033    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls podName:918210d0-9ac6-41c8-9573-3f5a5125e0fb nodeName:}" failed. No retries permitted until 2026-09-15 16:38:39.048014888 +0000 UTC m=+723.383969796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls") pod "isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" (UID: "918210d0-9ac6-41c8-9573-3f5a5125e0fb") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 15 16:38:38.548301 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:38:38.548054    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 15 16:38:38.548301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.548301 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:38:38.548106    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls podName:0628001b-71a5-4dce-bfca-3559f031c99f nodeName:}" failed. No retries permitted until 2026-09-15 16:38:39.048089558 +0000 UTC m=+723.384044495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls") pod "isvc-canary-rollback-predictor-6b76f97c96-l68pk" (UID: "0628001b-71a5-4dce-bfca-3559f031c99f") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 15 16:38:38.548301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548137    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.548301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbxc\" (UniqueName: \"kubernetes.io/projected/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-api-access-txbxc\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.548301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548213    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0628001b-71a5-4dce-bfca-3559f031c99f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.548683 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.548683 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548613    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0628001b-71a5-4dce-bfca-3559f031c99f-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.548683 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548652    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.548832 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548696    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.548832 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548800    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.548933 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.548873    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.557498 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.557471    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbxc\" (UniqueName: \"kubernetes.io/projected/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-api-access-txbxc\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:38.557498 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.557482    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xpgx\" (UniqueName: \"kubernetes.io/projected/0628001b-71a5-4dce-bfca-3559f031c99f-kube-api-access-7xpgx\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:38.851725 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.851690    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:38.854263 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:38.854238    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls\") pod \"isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:39.053733 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.053700    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:39.053899 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.053750    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:39.056332 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.056304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-l68pk\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:39.056484 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.056363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:39.101263 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.101229    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:38:39.291367 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.291328    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:38:39.324563 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.324525    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:38:39.376759 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.376734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"]
Sep 15 16:38:39.379442 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:38:39.379382    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e7ede59_a6f5_4e3b_872a_f0dbc27e623a.slice/crio-8b1abef2b63ae2680527a642752fd69d23ec2bd6ecf2c09f09b8f81fd6098ab4 WatchSource:0}: Error finding container 8b1abef2b63ae2680527a642752fd69d23ec2bd6ecf2c09f09b8f81fd6098ab4: Status 404 returned error can't find the container with id 8b1abef2b63ae2680527a642752fd69d23ec2bd6ecf2c09f09b8f81fd6098ab4
Sep 15 16:38:39.561832 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.561801    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"]
Sep 15 16:38:39.563875 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:38:39.563834    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918210d0_9ac6_41c8_9573_3f5a5125e0fb.slice/crio-e0da95267fa6d25b697dc897168e3be41ee8ae5646371b5a8d28442af09ca0ac WatchSource:0}: Error finding container e0da95267fa6d25b697dc897168e3be41ee8ae5646371b5a8d28442af09ca0ac: Status 404 returned error can't find the container with id e0da95267fa6d25b697dc897168e3be41ee8ae5646371b5a8d28442af09ca0ac
Sep 15 16:38:39.605105 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.605082    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"]
Sep 15 16:38:39.619523 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:38:39.619494    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0628001b_71a5_4dce_bfca_3559f031c99f.slice/crio-4574ef09dafd96c37c3531bfa2bf471c1030be5099ace9e39b171f169c9d53d3 WatchSource:0}: Error finding container 4574ef09dafd96c37c3531bfa2bf471c1030be5099ace9e39b171f169c9d53d3: Status 404 returned error can't find the container with id 4574ef09dafd96c37c3531bfa2bf471c1030be5099ace9e39b171f169c9d53d3
Sep 15 16:38:39.972804 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.972557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerStarted","Data":"7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe"}
Sep 15 16:38:39.972804 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.972640    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerStarted","Data":"8b1abef2b63ae2680527a642752fd69d23ec2bd6ecf2c09f09b8f81fd6098ab4"}
Sep 15 16:38:39.974521 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.974495    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" event={"ID":"0628001b-71a5-4dce-bfca-3559f031c99f","Type":"ContainerStarted","Data":"3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af"}
Sep 15 16:38:39.974725 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.974705    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" event={"ID":"0628001b-71a5-4dce-bfca-3559f031c99f","Type":"ContainerStarted","Data":"4574ef09dafd96c37c3531bfa2bf471c1030be5099ace9e39b171f169c9d53d3"}
Sep 15 16:38:39.976514 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.976489    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" event={"ID":"918210d0-9ac6-41c8-9573-3f5a5125e0fb","Type":"ContainerStarted","Data":"0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894"}
Sep 15 16:38:39.976623 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:39.976545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" event={"ID":"918210d0-9ac6-41c8-9573-3f5a5125e0fb","Type":"ContainerStarted","Data":"e0da95267fa6d25b697dc897168e3be41ee8ae5646371b5a8d28442af09ca0ac"}
Sep 15 16:38:43.995572 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:43.995485    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerID="7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe" exitCode=0
Sep 15 16:38:43.995948 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:43.995575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerDied","Data":"7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe"}
Sep 15 16:38:43.997657 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:43.997634    2564 generic.go:356] "Generic (PLEG): container finished" podID="0628001b-71a5-4dce-bfca-3559f031c99f" containerID="3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af" exitCode=0
Sep 15 16:38:43.997749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:43.997659    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" event={"ID":"0628001b-71a5-4dce-bfca-3559f031c99f","Type":"ContainerDied","Data":"3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af"}
Sep 15 16:38:43.999415 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:43.999375    2564 generic.go:356] "Generic (PLEG): container finished" podID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerID="0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894" exitCode=0
Sep 15 16:38:43.999473 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:38:43.999422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" event={"ID":"918210d0-9ac6-41c8-9573-3f5a5125e0fb","Type":"ContainerDied","Data":"0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894"}
Sep 15 16:39:04.184494 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:04.184458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerStarted","Data":"6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2"}
Sep 15 16:39:04.186422 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:04.186363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" event={"ID":"0628001b-71a5-4dce-bfca-3559f031c99f","Type":"ContainerStarted","Data":"127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738"}
Sep 15 16:39:04.188241 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:04.188216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" event={"ID":"918210d0-9ac6-41c8-9573-3f5a5125e0fb","Type":"ContainerStarted","Data":"d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab"}
Sep 15 16:39:07.205820 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.205788    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerStarted","Data":"7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd"}
Sep 15 16:39:07.207999 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.207962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" event={"ID":"0628001b-71a5-4dce-bfca-3559f031c99f","Type":"ContainerStarted","Data":"f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2"}
Sep 15 16:39:07.208161 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.208035    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:39:07.208161 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.208138    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:39:07.209667 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.209622    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:39:07.210384 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.210361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" event={"ID":"918210d0-9ac6-41c8-9573-3f5a5125e0fb","Type":"ContainerStarted","Data":"0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6"}
Sep 15 16:39:07.210618 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.210601    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:39:07.226921 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.226690    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podStartSLOduration=2.387657587 podStartE2EDuration="29.226668183s" totalImagesPullingTime="22.886603329s" totalInitContainerRuntime="3.952407267s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:38:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:39:07.225171893 +0000 UTC m=+751.561126823" watchObservedRunningTime="2026-09-15 16:39:07.226668183 +0000 UTC m=+751.562623120"
Sep 15 16:39:07.243617 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:07.242335    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podStartSLOduration=2.225302839 podStartE2EDuration="29.242308932s" totalImagesPullingTime="22.926386157s" totalInitContainerRuntime="4.090619936s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:38:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:39:07.241233936 +0000 UTC m=+751.577188864" watchObservedRunningTime="2026-09-15 16:39:07.242308932 +0000 UTC m=+751.578263864"
Sep 15 16:39:08.215260 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:08.215209    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:39:08.215779 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:08.215430    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:39:08.216526 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:08.216500    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:39:09.219473 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:09.219435    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:39:13.225307 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:13.225278    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:39:13.225836 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:13.225805    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:39:14.223844 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:14.223812    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:39:14.224329 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:14.224306    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:39:14.250378 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:14.250331    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerStarted","Data":"12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71"}
Sep 15 16:39:14.250845 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:14.250761    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:39:14.250845 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:14.250821    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:39:14.252237 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:14.252205    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:39:14.255691 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:14.255672    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:39:14.271627 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:14.271580    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podStartSLOduration=3.312006149 podStartE2EDuration="36.271566477s" totalImagesPullingTime="29.225665543s" totalInitContainerRuntime="3.733894785s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:38:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:39:14.269812621 +0000 UTC m=+758.605767554" watchObservedRunningTime="2026-09-15 16:39:14.271566477 +0000 UTC m=+758.607521407"
Sep 15 16:39:15.254680 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:15.254644    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:39:15.255208 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:15.254772    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:39:15.255650 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:15.255628    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:16.258899 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:16.258863    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:39:16.259298 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:16.259217    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:23.225958 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:23.225909    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:39:24.224334 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:24.224291    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:39:26.259251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:26.259203    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:39:26.259662 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:26.259645    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:33.226116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:33.226077    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:39:34.225017 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:34.224982    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:39:36.259060 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:36.259020    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:39:36.259586 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:36.259562    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:43.226756 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:43.226716    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:39:44.224601 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:44.224566    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:39:46.258968 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:46.258929    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:39:46.259429 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:46.259383    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:39:53.226551 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:53.226506    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:39:54.224583 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:54.224547    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:39:56.259276 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:56.259231    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:39:56.259780 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:39:56.259754    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:03.226750 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:03.226662    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:40:04.224321 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:04.224278    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:40:06.259698 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:06.259650    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:40:06.260132 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:06.259985    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:13.226250 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:13.226205    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 15 16:40:14.225000 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:14.224958    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:40:16.258931 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:16.258881    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:40:16.259352 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:16.259314    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:22.163997 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:22.163968    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:40:22.275170 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:22.275134    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"]
Sep 15 16:40:22.275424 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:22.275384    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container" containerID="cri-o://127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738" gracePeriod=30
Sep 15 16:40:22.275480 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:22.275441    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kube-rbac-proxy" containerID="cri-o://f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2" gracePeriod=30
Sep 15 16:40:22.276613 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:22.276593    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:40:22.535941 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:22.535900    2564 generic.go:356] "Generic (PLEG): container finished" podID="0628001b-71a5-4dce-bfca-3559f031c99f" containerID="f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2" exitCode=2
Sep 15 16:40:22.536096 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:22.535971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" event={"ID":"0628001b-71a5-4dce-bfca-3559f031c99f","Type":"ContainerDied","Data":"f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2"}
Sep 15 16:40:23.160595 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:23.160561    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:40:23.160786 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:23.160700    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:40:23.215859 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:23.215791    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 15 16:40:27.117187 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.117161    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:40:27.241563 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.241483    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0628001b-71a5-4dce-bfca-3559f031c99f-kserve-provision-location\") pod \"0628001b-71a5-4dce-bfca-3559f031c99f\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") "
Sep 15 16:40:27.241563 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.241525    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xpgx\" (UniqueName: \"kubernetes.io/projected/0628001b-71a5-4dce-bfca-3559f031c99f-kube-api-access-7xpgx\") pod \"0628001b-71a5-4dce-bfca-3559f031c99f\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") "
Sep 15 16:40:27.241563 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.241562    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls\") pod \"0628001b-71a5-4dce-bfca-3559f031c99f\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") "
Sep 15 16:40:27.241811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.241651    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-kube-rbac-proxy-sar-config\") pod \"0628001b-71a5-4dce-bfca-3559f031c99f\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") "
Sep 15 16:40:27.241811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.241699    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-cabundle-cert\") pod \"0628001b-71a5-4dce-bfca-3559f031c99f\" (UID: \"0628001b-71a5-4dce-bfca-3559f031c99f\") "
Sep 15 16:40:27.241928 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.241903    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0628001b-71a5-4dce-bfca-3559f031c99f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0628001b-71a5-4dce-bfca-3559f031c99f" (UID: "0628001b-71a5-4dce-bfca-3559f031c99f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:40:27.242051 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.242021    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0628001b-71a5-4dce-bfca-3559f031c99f" (UID: "0628001b-71a5-4dce-bfca-3559f031c99f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:27.242172 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.242050    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0628001b-71a5-4dce-bfca-3559f031c99f" (UID: "0628001b-71a5-4dce-bfca-3559f031c99f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:27.244060 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.244037    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0628001b-71a5-4dce-bfca-3559f031c99f" (UID: "0628001b-71a5-4dce-bfca-3559f031c99f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:40:27.244159 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.244037    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0628001b-71a5-4dce-bfca-3559f031c99f-kube-api-access-7xpgx" (OuterVolumeSpecName: "kube-api-access-7xpgx") pod "0628001b-71a5-4dce-bfca-3559f031c99f" (UID: "0628001b-71a5-4dce-bfca-3559f031c99f"). InnerVolumeSpecName "kube-api-access-7xpgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:40:27.343054 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.343016    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:27.343054 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.343051    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0628001b-71a5-4dce-bfca-3559f031c99f-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:27.343054 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.343065    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xpgx\" (UniqueName: \"kubernetes.io/projected/0628001b-71a5-4dce-bfca-3559f031c99f-kube-api-access-7xpgx\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:27.343322 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.343078    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0628001b-71a5-4dce-bfca-3559f031c99f-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:27.343322 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.343089    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0628001b-71a5-4dce-bfca-3559f031c99f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:27.559361 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.559326    2564 generic.go:356] "Generic (PLEG): container finished" podID="0628001b-71a5-4dce-bfca-3559f031c99f" containerID="127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738" exitCode=0
Sep 15 16:40:27.559549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.559418    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"
Sep 15 16:40:27.559549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.559428    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" event={"ID":"0628001b-71a5-4dce-bfca-3559f031c99f","Type":"ContainerDied","Data":"127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738"}
Sep 15 16:40:27.559549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.559486    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk" event={"ID":"0628001b-71a5-4dce-bfca-3559f031c99f","Type":"ContainerDied","Data":"4574ef09dafd96c37c3531bfa2bf471c1030be5099ace9e39b171f169c9d53d3"}
Sep 15 16:40:27.559549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.559514    2564 scope.go:117] "RemoveContainer" containerID="f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2"
Sep 15 16:40:27.568467 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.568449    2564 scope.go:117] "RemoveContainer" containerID="127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738"
Sep 15 16:40:27.576268 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.576248    2564 scope.go:117] "RemoveContainer" containerID="3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af"
Sep 15 16:40:27.581229 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.581207    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"]
Sep 15 16:40:27.584152 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.584133    2564 scope.go:117] "RemoveContainer" containerID="f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2"
Sep 15 16:40:27.584457 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:40:27.584433    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2\": container with ID starting with f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2 not found: ID does not exist" containerID="f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2"
Sep 15 16:40:27.584545 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.584467    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2"} err="failed to get container status \"f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2\": rpc error: code = NotFound desc = could not find container \"f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2\": container with ID starting with f552430867eda33889164a540045524c5ec2d738c16a562c38519957ecdd0fe2 not found: ID does not exist"
Sep 15 16:40:27.584545 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.584492    2564 scope.go:117] "RemoveContainer" containerID="127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738"
Sep 15 16:40:27.584915 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:40:27.584895    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738\": container with ID starting with 127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738 not found: ID does not exist" containerID="127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738"
Sep 15 16:40:27.584972 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.584944    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738"} err="failed to get container status \"127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738\": rpc error: code = NotFound desc = could not find container \"127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738\": container with ID starting with 127fc421ef1ea529218f1806950c450138958f20e6c7e0d7a54e586402931738 not found: ID does not exist"
Sep 15 16:40:27.584972 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.584966    2564 scope.go:117] "RemoveContainer" containerID="3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af"
Sep 15 16:40:27.585360 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:40:27.585333    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af\": container with ID starting with 3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af not found: ID does not exist" containerID="3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af"
Sep 15 16:40:27.585582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.585378    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af"} err="failed to get container status \"3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af\": rpc error: code = NotFound desc = could not find container \"3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af\": container with ID starting with 3db97d3c6c2258a6c563f9aa3ac49dfd6332c117b6ff641e38d1590bab5aa8af not found: ID does not exist"
Sep 15 16:40:27.587076 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:27.587057    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-l68pk"]
Sep 15 16:40:28.077987 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.077956    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"]
Sep 15 16:40:28.078276 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.078248    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" containerID="cri-o://d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab" gracePeriod=30
Sep 15 16:40:28.078380 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.078282    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kube-rbac-proxy" containerID="cri-o://0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6" gracePeriod=30
Sep 15 16:40:28.164233 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.164201    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" path="/var/lib/kubelet/pods/0628001b-71a5-4dce-bfca-3559f031c99f/volumes"
Sep 15 16:40:28.198046 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198026    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"]
Sep 15 16:40:28.198425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198409    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container"
Sep 15 16:40:28.198470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198427    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container"
Sep 15 16:40:28.198470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198436    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kube-rbac-proxy"
Sep 15 16:40:28.198470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198441    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kube-rbac-proxy"
Sep 15 16:40:28.198470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198458    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="storage-initializer"
Sep 15 16:40:28.198470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198463    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="storage-initializer"
Sep 15 16:40:28.198658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198517    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kserve-container"
Sep 15 16:40:28.198658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.198525    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0628001b-71a5-4dce-bfca-3559f031c99f" containerName="kube-rbac-proxy"
Sep 15 16:40:28.204242 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.203707    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.206223 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.206201    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 15 16:40:28.211507 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.211485    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"]
Sep 15 16:40:28.251934 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.251901    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tpwj\" (UniqueName: \"kubernetes.io/projected/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-api-access-9tpwj\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.252135 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.251971    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.252135 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.252035    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.252231 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.252135    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.252231 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.252191    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.352813 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.352708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.352813 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.352796    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tpwj\" (UniqueName: \"kubernetes.io/projected/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-api-access-9tpwj\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.353052 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.352856    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.353052 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.352882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.353052 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.352938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.353316 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.353293    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.353623 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.353605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.353683 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.353620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.355360 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.355343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.361378 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.361358    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tpwj\" (UniqueName: \"kubernetes.io/projected/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-api-access-9tpwj\") pod \"isvc-canary-rollback-predictor-6b76f97c96-7pl2d\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.515568 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.515528    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:28.568746 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.568685    2564 generic.go:356] "Generic (PLEG): container finished" podID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerID="0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6" exitCode=2
Sep 15 16:40:28.568746 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.568717    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" event={"ID":"918210d0-9ac6-41c8-9573-3f5a5125e0fb","Type":"ContainerDied","Data":"0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6"}
Sep 15 16:40:28.788087 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:28.788051    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"]
Sep 15 16:40:28.790353 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:40:28.790332    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbccd24fd_f5f7_40fa_b5fc_a2b091e68da2.slice/crio-a6208fafcae414ac33d200e358d9114434464201465d8d241e13b35796cb919c WatchSource:0}: Error finding container a6208fafcae414ac33d200e358d9114434464201465d8d241e13b35796cb919c: Status 404 returned error can't find the container with id a6208fafcae414ac33d200e358d9114434464201465d8d241e13b35796cb919c
Sep 15 16:40:29.220843 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:29.220748    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Sep 15 16:40:29.576341 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:29.576306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" event={"ID":"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2","Type":"ContainerStarted","Data":"15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa"}
Sep 15 16:40:29.576498 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:29.576364    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" event={"ID":"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2","Type":"ContainerStarted","Data":"a6208fafcae414ac33d200e358d9114434464201465d8d241e13b35796cb919c"}
Sep 15 16:40:32.160029 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.159989    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 15 16:40:32.921013 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.920990    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:40:32.993547 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.993509    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-rbac-proxy-sar-config\") pod \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") "
Sep 15 16:40:32.993738 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.993566    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kserve-provision-location\") pod \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") "
Sep 15 16:40:32.993738 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.993612    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txbxc\" (UniqueName: \"kubernetes.io/projected/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-api-access-txbxc\") pod \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") "
Sep 15 16:40:32.993738 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.993664    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-cabundle-cert\") pod \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") "
Sep 15 16:40:32.993738 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.993691    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls\") pod \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\" (UID: \"918210d0-9ac6-41c8-9573-3f5a5125e0fb\") "
Sep 15 16:40:32.993952 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.993925    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "918210d0-9ac6-41c8-9573-3f5a5125e0fb" (UID: "918210d0-9ac6-41c8-9573-3f5a5125e0fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:40:32.993994 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.993949    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "918210d0-9ac6-41c8-9573-3f5a5125e0fb" (UID: "918210d0-9ac6-41c8-9573-3f5a5125e0fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:32.994028 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.994015    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "918210d0-9ac6-41c8-9573-3f5a5125e0fb" (UID: "918210d0-9ac6-41c8-9573-3f5a5125e0fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:40:32.995996 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.995967    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "918210d0-9ac6-41c8-9573-3f5a5125e0fb" (UID: "918210d0-9ac6-41c8-9573-3f5a5125e0fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:40:32.995996 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:32.995973    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-api-access-txbxc" (OuterVolumeSpecName: "kube-api-access-txbxc") pod "918210d0-9ac6-41c8-9573-3f5a5125e0fb" (UID: "918210d0-9ac6-41c8-9573-3f5a5125e0fb"). InnerVolumeSpecName "kube-api-access-txbxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:40:33.094273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.094235    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:33.094273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.094264    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/918210d0-9ac6-41c8-9573-3f5a5125e0fb-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:33.094273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.094273    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:33.094273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.094284    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:33.094548 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.094293    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txbxc\" (UniqueName: \"kubernetes.io/projected/918210d0-9ac6-41c8-9573-3f5a5125e0fb-kube-api-access-txbxc\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:40:33.147693 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.147665    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"]
Sep 15 16:40:33.147955 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.147934    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" containerID="cri-o://6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2" gracePeriod=30
Sep 15 16:40:33.148049 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.148020    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy" containerID="cri-o://7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd" gracePeriod=30
Sep 15 16:40:33.148126 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.148023    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" containerID="cri-o://12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71" gracePeriod=30
Sep 15 16:40:33.160662 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.160637    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:33.403957 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.403925    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"]
Sep 15 16:40:33.404295 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.404282    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container"
Sep 15 16:40:33.404345 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.404298    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container"
Sep 15 16:40:33.404345 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.404308    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kube-rbac-proxy"
Sep 15 16:40:33.404345 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.404314    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kube-rbac-proxy"
Sep 15 16:40:33.404345 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.404340    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="storage-initializer"
Sep 15 16:40:33.404492 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.404346    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="storage-initializer"
Sep 15 16:40:33.404492 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.404420    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kserve-container"
Sep 15 16:40:33.404492 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.404430    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerName="kube-rbac-proxy"
Sep 15 16:40:33.407413 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.407383    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.409873 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.409853    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e2542-predictor-serving-cert\""
Sep 15 16:40:33.409992 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.409854    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-e2542-kube-rbac-proxy-sar-config\""
Sep 15 16:40:33.415863 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.415840    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"]
Sep 15 16:40:33.496999 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.496952    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a771202e-8252-4280-a4d6-5331082dd590-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.497181 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.497068    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.497181 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.497139    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771202e-8252-4280-a4d6-5331082dd590-proxy-tls\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.497181 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.497175    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvbd\" (UniqueName: \"kubernetes.io/projected/a771202e-8252-4280-a4d6-5331082dd590-kube-api-access-vxvbd\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.497309 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.497200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.597890 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.597804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.597890 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.597871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771202e-8252-4280-a4d6-5331082dd590-proxy-tls\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.598116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.597911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvbd\" (UniqueName: \"kubernetes.io/projected/a771202e-8252-4280-a4d6-5331082dd590-kube-api-access-vxvbd\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.598215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.598191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.598342 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.598255    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a771202e-8252-4280-a4d6-5331082dd590-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.598778 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.598746    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a771202e-8252-4280-a4d6-5331082dd590-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.598886 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.598832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-cabundle-cert\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.598988 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.598970    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.600074 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.600052    2564 generic.go:356] "Generic (PLEG): container finished" podID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" containerID="d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab" exitCode=0
Sep 15 16:40:33.600163 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.600147    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"
Sep 15 16:40:33.600251 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.600141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" event={"ID":"918210d0-9ac6-41c8-9573-3f5a5125e0fb","Type":"ContainerDied","Data":"d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab"}
Sep 15 16:40:33.600350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.600336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn" event={"ID":"918210d0-9ac6-41c8-9573-3f5a5125e0fb","Type":"ContainerDied","Data":"e0da95267fa6d25b697dc897168e3be41ee8ae5646371b5a8d28442af09ca0ac"}
Sep 15 16:40:33.600430 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.600381    2564 scope.go:117] "RemoveContainer" containerID="0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6"
Sep 15 16:40:33.601138 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.601114    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771202e-8252-4280-a4d6-5331082dd590-proxy-tls\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.602073 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.602054    2564 generic.go:356] "Generic (PLEG): container finished" podID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerID="15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa" exitCode=0
Sep 15 16:40:33.602163 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.602125    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" event={"ID":"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2","Type":"ContainerDied","Data":"15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa"}
Sep 15 16:40:33.605370 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.605351    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerID="7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd" exitCode=2
Sep 15 16:40:33.605500 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.605411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerDied","Data":"7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd"}
Sep 15 16:40:33.606610 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.606588    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvbd\" (UniqueName: \"kubernetes.io/projected/a771202e-8252-4280-a4d6-5331082dd590-kube-api-access-vxvbd\") pod \"isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.609773 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.609757    2564 scope.go:117] "RemoveContainer" containerID="d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab"
Sep 15 16:40:33.620274 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.620252    2564 scope.go:117] "RemoveContainer" containerID="0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894"
Sep 15 16:40:33.630820 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.630799    2564 scope.go:117] "RemoveContainer" containerID="0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6"
Sep 15 16:40:33.631117 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:40:33.631093    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6\": container with ID starting with 0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6 not found: ID does not exist" containerID="0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6"
Sep 15 16:40:33.631191 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.631127    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6"} err="failed to get container status \"0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6\": rpc error: code = NotFound desc = could not find container \"0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6\": container with ID starting with 0aca68e5e65706675209b61c61d26c467babb634e0b5f4186dc4f449d6fe70f6 not found: ID does not exist"
Sep 15 16:40:33.631191 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.631145    2564 scope.go:117] "RemoveContainer" containerID="d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab"
Sep 15 16:40:33.631451 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:40:33.631434    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab\": container with ID starting with d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab not found: ID does not exist" containerID="d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab"
Sep 15 16:40:33.631488 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.631458    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab"} err="failed to get container status \"d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab\": rpc error: code = NotFound desc = could not find container \"d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab\": container with ID starting with d8918eff28a352b1f66ef4738d907ebbf806db65b9bd87fd5244343f4178c5ab not found: ID does not exist"
Sep 15 16:40:33.631488 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.631472    2564 scope.go:117] "RemoveContainer" containerID="0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894"
Sep 15 16:40:33.631718 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:40:33.631703    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894\": container with ID starting with 0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894 not found: ID does not exist" containerID="0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894"
Sep 15 16:40:33.631758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.631726    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894"} err="failed to get container status \"0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894\": rpc error: code = NotFound desc = could not find container \"0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894\": container with ID starting with 0d95ea36d70a8f585d2e9642047b0760195828c1cf92a02a1720083ce3011894 not found: ID does not exist"
Sep 15 16:40:33.634128 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.634105    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"]
Sep 15 16:40:33.637826 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.637809    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-k6fxn"]
Sep 15 16:40:33.718488 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.718464    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:40:33.996060 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:33.996036    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"]
Sep 15 16:40:33.998205 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:40:33.998180    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda771202e_8252_4280_a4d6_5331082dd590.slice/crio-9d91638d40f0d160f118621479dae3e5f174fdda00d5e412e433ef2b7792a524 WatchSource:0}: Error finding container 9d91638d40f0d160f118621479dae3e5f174fdda00d5e412e433ef2b7792a524: Status 404 returned error can't find the container with id 9d91638d40f0d160f118621479dae3e5f174fdda00d5e412e433ef2b7792a524
Sep 15 16:40:34.164564 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.164538    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="918210d0-9ac6-41c8-9573-3f5a5125e0fb" path="/var/lib/kubelet/pods/918210d0-9ac6-41c8-9573-3f5a5125e0fb/volumes"
Sep 15 16:40:34.251942 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.251893    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 15 16:40:34.614115 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.614026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" event={"ID":"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2","Type":"ContainerStarted","Data":"3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc"}
Sep 15 16:40:34.614115 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.614089    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" event={"ID":"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2","Type":"ContainerStarted","Data":"51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4"}
Sep 15 16:40:34.614461 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.614432    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:34.614595 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.614497    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:34.615729 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.615708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" event={"ID":"a771202e-8252-4280-a4d6-5331082dd590","Type":"ContainerStarted","Data":"ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f"}
Sep 15 16:40:34.615838 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.615754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" event={"ID":"a771202e-8252-4280-a4d6-5331082dd590","Type":"ContainerStarted","Data":"9d91638d40f0d160f118621479dae3e5f174fdda00d5e412e433ef2b7792a524"}
Sep 15 16:40:34.616218 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.616194    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:40:34.631581 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:34.631538    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podStartSLOduration=2.717418625 podStartE2EDuration="6.631519257s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.914100632s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:40:34.630463817 +0000 UTC m=+838.966418750" watchObservedRunningTime="2026-09-15 16:40:34.631519257 +0000 UTC m=+838.967474188"
Sep 15 16:40:35.620239 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:35.620201    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:40:38.635301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:38.635265    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerID="6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2" exitCode=0
Sep 15 16:40:38.635707 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:38.635341    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerDied","Data":"6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2"}
Sep 15 16:40:38.636877 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:38.636858    2564 generic.go:356] "Generic (PLEG): container finished" podID="a771202e-8252-4280-a4d6-5331082dd590" containerID="ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f" exitCode=0
Sep 15 16:40:38.636971 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:38.636886    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" event={"ID":"a771202e-8252-4280-a4d6-5331082dd590","Type":"ContainerDied","Data":"ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f"}
Sep 15 16:40:38.638211 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:38.638196    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:40:39.251628 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:39.251165    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 15 16:40:40.626281 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:40.626241    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:40:40.626869 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:40.626837    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:40:43.160433 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:43.160357    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:40:43.160893 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:43.160785    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:44.251901 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:44.251853    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 15 16:40:44.252278 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:44.251995    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:40:49.251058 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:49.251008    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 15 16:40:50.626819 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:50.626770    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:40:53.160675 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:53.160633    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:40:53.161117 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:53.160997    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:40:53.161117 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:53.161114    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:40:54.251800 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:54.251758    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 15 16:40:59.251552 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:40:59.251500    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 15 16:41:00.626749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:00.626713    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:41:00.745362 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:00.745322    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" event={"ID":"a771202e-8252-4280-a4d6-5331082dd590","Type":"ContainerStarted","Data":"d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be"}
Sep 15 16:41:00.745525 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:00.745409    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" event={"ID":"a771202e-8252-4280-a4d6-5331082dd590","Type":"ContainerStarted","Data":"ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37"}
Sep 15 16:41:00.745691 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:00.745676    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:41:00.768517 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:00.768465    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podStartSLOduration=2.4253467300000002 podStartE2EDuration="27.768446307s" totalImagesPullingTime="21.288543301s" totalInitContainerRuntime="4.054556276s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:41:00.766227292 +0000 UTC m=+865.102182224" watchObservedRunningTime="2026-09-15 16:41:00.768446307 +0000 UTC m=+865.104401237"
Sep 15 16:41:01.750876 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:01.750834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:41:01.752273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:01.752241    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:41:02.755338 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:02.755291    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:41:03.161011 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.160951    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 15 16:41:03.161124 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.161046    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent" probeResult="failure" output="Get \"http://10.134.0.22:9081/\": dial tcp 10.134.0.22:9081: connect: connection refused"
Sep 15 16:41:03.161188 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.161123    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:41:03.296215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.296191    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:41:03.362793 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.362766    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjh6t\" (UniqueName: \"kubernetes.io/projected/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-api-access-tjh6t\") pod \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") "
Sep 15 16:41:03.362970 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.362814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kserve-provision-location\") pod \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") "
Sep 15 16:41:03.362970 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.362835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-cabundle-cert\") pod \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") "
Sep 15 16:41:03.362970 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.362886    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-rbac-proxy-sar-config\") pod \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") "
Sep 15 16:41:03.362970 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.362935    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls\") pod \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\" (UID: \"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a\") "
Sep 15 16:41:03.363192 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.363142    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" (UID: "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:41:03.363275 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.363251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" (UID: "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:41:03.363356 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.363332    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" (UID: "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:41:03.365158 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.365133    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-api-access-tjh6t" (OuterVolumeSpecName: "kube-api-access-tjh6t") pod "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" (UID: "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a"). InnerVolumeSpecName "kube-api-access-tjh6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:41:03.365280 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.365194    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" (UID: "2e7ede59-a6f5-4e3b-872a-f0dbc27e623a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:41:03.464041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.463959    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:41:03.464041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.463987    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:41:03.464041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.463996    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjh6t\" (UniqueName: \"kubernetes.io/projected/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kube-api-access-tjh6t\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:41:03.464041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.464005    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:41:03.464041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.464016    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:41:03.761085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.761055    2564 generic.go:356] "Generic (PLEG): container finished" podID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerID="12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71" exitCode=0
Sep 15 16:41:03.761502 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.761114    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"
Sep 15 16:41:03.761502 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.761139    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerDied","Data":"12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71"}
Sep 15 16:41:03.761502 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.761205    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h" event={"ID":"2e7ede59-a6f5-4e3b-872a-f0dbc27e623a","Type":"ContainerDied","Data":"8b1abef2b63ae2680527a642752fd69d23ec2bd6ecf2c09f09b8f81fd6098ab4"}
Sep 15 16:41:03.761502 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.761235    2564 scope.go:117] "RemoveContainer" containerID="12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71"
Sep 15 16:41:03.770543 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.770524    2564 scope.go:117] "RemoveContainer" containerID="7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd"
Sep 15 16:41:03.778035 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.778012    2564 scope.go:117] "RemoveContainer" containerID="6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2"
Sep 15 16:41:03.783462 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.783440    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"]
Sep 15 16:41:03.786331 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.786311    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-d4131-predictor-7b9fdb8c4f-ms25h"]
Sep 15 16:41:03.786591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.786570    2564 scope.go:117] "RemoveContainer" containerID="7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe"
Sep 15 16:41:03.794247 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.794228    2564 scope.go:117] "RemoveContainer" containerID="12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71"
Sep 15 16:41:03.794564 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:41:03.794542    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71\": container with ID starting with 12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71 not found: ID does not exist" containerID="12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71"
Sep 15 16:41:03.794670 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.794572    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71"} err="failed to get container status \"12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71\": rpc error: code = NotFound desc = could not find container \"12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71\": container with ID starting with 12e5e3b152371f2fd297d9319033c38555c15788efea9cb3acd17fd3dc818d71 not found: ID does not exist"
Sep 15 16:41:03.794670 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.794591    2564 scope.go:117] "RemoveContainer" containerID="7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd"
Sep 15 16:41:03.794889 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:41:03.794870    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd\": container with ID starting with 7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd not found: ID does not exist" containerID="7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd"
Sep 15 16:41:03.794935 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.794895    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd"} err="failed to get container status \"7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd\": rpc error: code = NotFound desc = could not find container \"7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd\": container with ID starting with 7be7e4e42a352ffdf6a1a1ab00eeb9e46de4ed4239b466f598a2ce416454c0dd not found: ID does not exist"
Sep 15 16:41:03.794935 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.794911    2564 scope.go:117] "RemoveContainer" containerID="6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2"
Sep 15 16:41:03.795146 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:41:03.795129    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2\": container with ID starting with 6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2 not found: ID does not exist" containerID="6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2"
Sep 15 16:41:03.795200 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.795147    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2"} err="failed to get container status \"6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2\": rpc error: code = NotFound desc = could not find container \"6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2\": container with ID starting with 6b83c68cd8acf0e0ea5d37ceaa3d24e17c634b2d8e08e1f6eb7d2b6ba5ec7fa2 not found: ID does not exist"
Sep 15 16:41:03.795200 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.795158    2564 scope.go:117] "RemoveContainer" containerID="7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe"
Sep 15 16:41:03.795413 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:41:03.795382    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe\": container with ID starting with 7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe not found: ID does not exist" containerID="7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe"
Sep 15 16:41:03.795465 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:03.795420    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe"} err="failed to get container status \"7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe\": rpc error: code = NotFound desc = could not find container \"7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe\": container with ID starting with 7a1c7e95f78032c943cc07987293cfe095f3a8cfaf580efc2d5c8716efd3e2fe not found: ID does not exist"
Sep 15 16:41:04.164085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:04.164007    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" path="/var/lib/kubelet/pods/2e7ede59-a6f5-4e3b-872a-f0dbc27e623a/volumes"
Sep 15 16:41:07.760198 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:07.760169    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:41:07.760686 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:07.760660    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:41:10.627462 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:10.627427    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:41:17.760782 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:17.760736    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:41:20.627121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:20.627061    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:41:27.761623 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:27.761534    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:41:30.627695 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:30.627648    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:41:37.761015 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:37.760970    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:41:40.626756 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:40.626714    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:41:45.160359 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:45.160314    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Sep 15 16:41:47.760850 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:47.760811    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:41:55.161591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:55.161560    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:41:57.761324 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:57.761280    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Sep 15 16:41:58.250343 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:58.250257    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"]
Sep 15 16:41:58.250625 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:58.250577    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container" containerID="cri-o://51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4" gracePeriod=30
Sep 15 16:41:58.250763 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:58.250618    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kube-rbac-proxy" containerID="cri-o://3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc" gracePeriod=30
Sep 15 16:41:58.982146 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:58.982114    2564 generic.go:356] "Generic (PLEG): container finished" podID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerID="3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc" exitCode=2
Sep 15 16:41:58.982561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:41:58.982187    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" event={"ID":"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2","Type":"ContainerDied","Data":"3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc"}
Sep 15 16:42:00.621361 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:00.621321    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Sep 15 16:42:02.888747 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.888723    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:42:02.960769 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.960731    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kserve-provision-location\") pod \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") "
Sep 15 16:42:02.960960 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.960779    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-cabundle-cert\") pod \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") "
Sep 15 16:42:02.960960 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.960823    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-rbac-proxy-sar-config\") pod \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") "
Sep 15 16:42:02.960960 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.960854    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tpwj\" (UniqueName: \"kubernetes.io/projected/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-api-access-9tpwj\") pod \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") "
Sep 15 16:42:02.960960 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.960892    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-proxy-tls\") pod \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\" (UID: \"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2\") "
Sep 15 16:42:02.961223 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.961198    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" (UID: "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:42:02.961271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.961196    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" (UID: "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:42:02.961271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.961259    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" (UID: "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:42:02.963180 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.963154    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" (UID: "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:42:02.963290 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.963189    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-api-access-9tpwj" (OuterVolumeSpecName: "kube-api-access-9tpwj") pod "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" (UID: "bccd24fd-f5f7-40fa-b5fc-a2b091e68da2"). InnerVolumeSpecName "kube-api-access-9tpwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:42:03.000071 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:02.999988    2564 generic.go:356] "Generic (PLEG): container finished" podID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerID="51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4" exitCode=0
Sep 15 16:42:03.000071 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.000059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" event={"ID":"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2","Type":"ContainerDied","Data":"51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4"}
Sep 15 16:42:03.000284 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.000104    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d" event={"ID":"bccd24fd-f5f7-40fa-b5fc-a2b091e68da2","Type":"ContainerDied","Data":"a6208fafcae414ac33d200e358d9114434464201465d8d241e13b35796cb919c"}
Sep 15 16:42:03.000284 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.000111    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"
Sep 15 16:42:03.000284 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.000130    2564 scope.go:117] "RemoveContainer" containerID="3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc"
Sep 15 16:42:03.008674 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.008659    2564 scope.go:117] "RemoveContainer" containerID="51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4"
Sep 15 16:42:03.016075 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.016059    2564 scope.go:117] "RemoveContainer" containerID="15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa"
Sep 15 16:42:03.021140 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.021113    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"]
Sep 15 16:42:03.024000 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.023976    2564 scope.go:117] "RemoveContainer" containerID="3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc"
Sep 15 16:42:03.024318 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:03.024299    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc\": container with ID starting with 3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc not found: ID does not exist" containerID="3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc"
Sep 15 16:42:03.024372 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.024329    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc"} err="failed to get container status \"3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc\": rpc error: code = NotFound desc = could not find container \"3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc\": container with ID starting with 3fe863af88ca28422d0363e0c4e599db6607afc17457c0fe8202fe3054efdcdc not found: ID does not exist"
Sep 15 16:42:03.024372 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.024347    2564 scope.go:117] "RemoveContainer" containerID="51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4"
Sep 15 16:42:03.024697 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:03.024674    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4\": container with ID starting with 51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4 not found: ID does not exist" containerID="51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4"
Sep 15 16:42:03.024777 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.024703    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4"} err="failed to get container status \"51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4\": rpc error: code = NotFound desc = could not find container \"51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4\": container with ID starting with 51c106abcb595753cf4cc377d11dbaf9c519df29bd261341afd27d59f34955a4 not found: ID does not exist"
Sep 15 16:42:03.024777 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.024725    2564 scope.go:117] "RemoveContainer" containerID="15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa"
Sep 15 16:42:03.024982 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:03.024968    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa\": container with ID starting with 15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa not found: ID does not exist" containerID="15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa"
Sep 15 16:42:03.025024 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.024985    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa"} err="failed to get container status \"15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa\": rpc error: code = NotFound desc = could not find container \"15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa\": container with ID starting with 15978f22baf42235db1d48b7c8ac723c5adba50fb769b94ad1703ac48a26cbfa not found: ID does not exist"
Sep 15 16:42:03.025358 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.025341    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-7pl2d"]
Sep 15 16:42:03.061576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.061539    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:03.061576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.061573    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:03.061576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.061583    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:03.061769 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.061593    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:03.061769 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.061601    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tpwj\" (UniqueName: \"kubernetes.io/projected/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2-kube-api-access-9tpwj\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:03.441796 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.441761    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"]
Sep 15 16:42:03.442068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442056    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="storage-initializer"
Sep 15 16:42:03.442114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442070    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="storage-initializer"
Sep 15 16:42:03.442114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442081    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy"
Sep 15 16:42:03.442114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442086    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy"
Sep 15 16:42:03.442114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442094    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container"
Sep 15 16:42:03.442114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442099    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container"
Sep 15 16:42:03.442114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442108    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent"
Sep 15 16:42:03.442114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442113    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442122    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442127    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442136    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kube-rbac-proxy"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442141    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kube-rbac-proxy"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442149    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="storage-initializer"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442154    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="storage-initializer"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442199    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kube-rbac-proxy"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442208    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kserve-container"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442216    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="agent"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442221    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" containerName="kserve-container"
Sep 15 16:42:03.442315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.442227    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e7ede59-a6f5-4e3b-872a-f0dbc27e623a" containerName="kube-rbac-proxy"
Sep 15 16:42:03.445113 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.445097    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.447791 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.447772    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 15 16:42:03.447919 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.447775    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 15 16:42:03.459790 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.459765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"]
Sep 15 16:42:03.462947 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.462922    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"]
Sep 15 16:42:03.465252 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.465236    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.465460 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.465440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.465549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.465472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.465549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.465497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvgqh\" (UniqueName: \"kubernetes.io/projected/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-api-access-lvgqh\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.465665 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.465554    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.465665 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.465614    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.467458 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.467439    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 15 16:42:03.477136 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.477113    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"]
Sep 15 16:42:03.566808 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.566775    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.566994 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.566822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.566994 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.566839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.566994 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.566964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.567152 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567004    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj89g\" (UniqueName: \"kubernetes.io/projected/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-api-access-rj89g\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.567152 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvgqh\" (UniqueName: \"kubernetes.io/projected/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-api-access-lvgqh\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.567152 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.567152 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.567381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567158    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.567381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567188    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.567598 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567573    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.567748 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567729    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.567748 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.567741    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.569344 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.569326    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.574749 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.574726    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvgqh\" (UniqueName: \"kubernetes.io/projected/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-api-access-lvgqh\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-jh56k\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:03.667735 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.667692    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.667907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.667749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj89g\" (UniqueName: \"kubernetes.io/projected/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-api-access-rj89g\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.667907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.667794    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.667907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.667810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.667907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.667835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.667907 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:03.667853    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 15 16:42:03.668096 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:03.667916    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls podName:51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2 nodeName:}" failed. No retries permitted until 2026-09-15 16:42:04.167897691 +0000 UTC m=+928.503852618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls") pod "isvc-canary-stop-predictor-7f6f9c568d-4bzqx" (UID: "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 15 16:42:03.668271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.668250    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.668575 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.668559    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.668619 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.668592    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.676082 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.676053    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj89g\" (UniqueName: \"kubernetes.io/projected/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-api-access-rj89g\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:03.755242 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:03.755207    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:04.024770 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:04.024669    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"]
Sep 15 16:42:04.026935 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:42:04.026907    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b3dd63_751d_494b_b7b7_b6ef0f6a63f9.slice/crio-556cf8a5874bef3f7985a028b668dd94893a6a1129cffd15dcd1c8d135223cea WatchSource:0}: Error finding container 556cf8a5874bef3f7985a028b668dd94893a6a1129cffd15dcd1c8d135223cea: Status 404 returned error can't find the container with id 556cf8a5874bef3f7985a028b668dd94893a6a1129cffd15dcd1c8d135223cea
Sep 15 16:42:04.166151 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:04.166117    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bccd24fd-f5f7-40fa-b5fc-a2b091e68da2" path="/var/lib/kubelet/pods/bccd24fd-f5f7-40fa-b5fc-a2b091e68da2/volumes"
Sep 15 16:42:04.171894 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:04.171866    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:04.174318 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:04.174295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-4bzqx\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:04.375780 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:04.375690    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:04.646591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:04.646434    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"]
Sep 15 16:42:04.649019 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:42:04.648984    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aba90f_0bfa_49c2_ae3b_1fb7d3ea5bd2.slice/crio-572e971fb80ae368af7a10bc0e2aead59b5b7f5fd5751c09b28629b74796a885 WatchSource:0}: Error finding container 572e971fb80ae368af7a10bc0e2aead59b5b7f5fd5751c09b28629b74796a885: Status 404 returned error can't find the container with id 572e971fb80ae368af7a10bc0e2aead59b5b7f5fd5751c09b28629b74796a885
Sep 15 16:42:05.011375 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:05.011336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" event={"ID":"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2","Type":"ContainerStarted","Data":"97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d"}
Sep 15 16:42:05.011579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:05.011421    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" event={"ID":"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2","Type":"ContainerStarted","Data":"572e971fb80ae368af7a10bc0e2aead59b5b7f5fd5751c09b28629b74796a885"}
Sep 15 16:42:05.013146 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:05.013120    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" event={"ID":"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9","Type":"ContainerStarted","Data":"66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf"}
Sep 15 16:42:05.013260 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:05.013179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" event={"ID":"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9","Type":"ContainerStarted","Data":"556cf8a5874bef3f7985a028b668dd94893a6a1129cffd15dcd1c8d135223cea"}
Sep 15 16:42:07.762041 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:07.762009    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:42:08.026999 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:08.026925    2564 generic.go:356] "Generic (PLEG): container finished" podID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerID="66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf" exitCode=0
Sep 15 16:42:08.026999 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:08.026961    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" event={"ID":"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9","Type":"ContainerDied","Data":"66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf"}
Sep 15 16:42:09.032552 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:09.032517    2564 generic.go:356] "Generic (PLEG): container finished" podID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerID="97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d" exitCode=0
Sep 15 16:42:09.033019 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:09.032594    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" event={"ID":"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2","Type":"ContainerDied","Data":"97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d"}
Sep 15 16:42:09.034857 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:09.034838    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" event={"ID":"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9","Type":"ContainerStarted","Data":"e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9"}
Sep 15 16:42:09.034940 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:09.034879    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" event={"ID":"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9","Type":"ContainerStarted","Data":"0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061"}
Sep 15 16:42:09.036304 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:09.036288    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:09.036425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:09.036323    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:09.037356 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:09.037337    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:42:09.068277 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:09.068232    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podStartSLOduration=2.233037108 podStartE2EDuration="6.068216368s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83517926s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:42:09.066319087 +0000 UTC m=+933.402274018" watchObservedRunningTime="2026-09-15 16:42:09.068216368 +0000 UTC m=+933.404171299"
Sep 15 16:42:10.041042 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:10.041009    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" event={"ID":"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2","Type":"ContainerStarted","Data":"01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8"}
Sep 15 16:42:10.041435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:10.041097    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" event={"ID":"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2","Type":"ContainerStarted","Data":"2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69"}
Sep 15 16:42:10.041435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:10.041302    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:42:10.041556 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:10.041536    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:10.041735 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:10.041707    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:10.042692 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:10.042662    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:42:10.059549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:10.059505    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podStartSLOduration=3.272664325 podStartE2EDuration="7.059492823s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786828498s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:42:10.058467107 +0000 UTC m=+934.394422038" watchObservedRunningTime="2026-09-15 16:42:10.059492823 +0000 UTC m=+934.395447753"
Sep 15 16:42:11.046348 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:11.046303    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:42:13.325299 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.325269    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"]
Sep 15 16:42:13.327428 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.327412    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:13.329886 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.329866    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-e2542-serving-cert\""
Sep 15 16:42:13.330014 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.329998    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-e2542-kube-rbac-proxy-sar-config\""
Sep 15 16:42:13.339529 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.339509    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"]
Sep 15 16:42:13.346201 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.346178    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls\") pod \"model-chainer-raw-e2542-7fdc9d5f94-mpc8d\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") " pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:13.346320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.346246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb01404-179e-4f58-a3f6-662a72e23ff2-openshift-service-ca-bundle\") pod \"model-chainer-raw-e2542-7fdc9d5f94-mpc8d\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") " pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:13.446958 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.446926    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb01404-179e-4f58-a3f6-662a72e23ff2-openshift-service-ca-bundle\") pod \"model-chainer-raw-e2542-7fdc9d5f94-mpc8d\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") " pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:13.446958 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.446969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls\") pod \"model-chainer-raw-e2542-7fdc9d5f94-mpc8d\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") " pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:13.447229 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:13.447078    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-e2542-serving-cert: secret "model-chainer-raw-e2542-serving-cert" not found
Sep 15 16:42:13.447229 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:13.447150    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls podName:cdb01404-179e-4f58-a3f6-662a72e23ff2 nodeName:}" failed. No retries permitted until 2026-09-15 16:42:13.947129244 +0000 UTC m=+938.283084154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls") pod "model-chainer-raw-e2542-7fdc9d5f94-mpc8d" (UID: "cdb01404-179e-4f58-a3f6-662a72e23ff2") : secret "model-chainer-raw-e2542-serving-cert" not found
Sep 15 16:42:13.447667 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.447644    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb01404-179e-4f58-a3f6-662a72e23ff2-openshift-service-ca-bundle\") pod \"model-chainer-raw-e2542-7fdc9d5f94-mpc8d\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") " pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:13.952446 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.952405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls\") pod \"model-chainer-raw-e2542-7fdc9d5f94-mpc8d\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") " pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:13.954902 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:13.954877    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls\") pod \"model-chainer-raw-e2542-7fdc9d5f94-mpc8d\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") " pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:14.237207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:14.237132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:14.504438 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:14.504413    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"]
Sep 15 16:42:14.506917 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:42:14.506893    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb01404_179e_4f58_a3f6_662a72e23ff2.slice/crio-c38938f63ffcabc138f4b149e48c31c41b96f450e135282038065c4cec954317 WatchSource:0}: Error finding container c38938f63ffcabc138f4b149e48c31c41b96f450e135282038065c4cec954317: Status 404 returned error can't find the container with id c38938f63ffcabc138f4b149e48c31c41b96f450e135282038065c4cec954317
Sep 15 16:42:15.046300 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:15.046274    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:42:15.046852 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:15.046823    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:42:15.064592 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:15.064570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" event={"ID":"cdb01404-179e-4f58-a3f6-662a72e23ff2","Type":"ContainerStarted","Data":"c38938f63ffcabc138f4b149e48c31c41b96f450e135282038065c4cec954317"}
Sep 15 16:42:16.052143 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:16.052112    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:42:16.052926 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:16.052891    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:42:17.076175 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:17.076137    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" event={"ID":"cdb01404-179e-4f58-a3f6-662a72e23ff2","Type":"ContainerStarted","Data":"dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b"}
Sep 15 16:42:17.076657 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:17.076279    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:17.092822 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:17.092777    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" podStartSLOduration=2.159178857 podStartE2EDuration="4.092761543s" totalImagesPullingTime="1.933582686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:42:17.090848047 +0000 UTC m=+941.426802972" watchObservedRunningTime="2026-09-15 16:42:17.092761543 +0000 UTC m=+941.428716473"
Sep 15 16:42:23.086043 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.086016    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:23.381970 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.381894    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"]
Sep 15 16:42:23.382112 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.382079    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542" containerID="cri-o://dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b" gracePeriod=30
Sep 15 16:42:23.530311 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.530281    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"]
Sep 15 16:42:23.530560 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.530533    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container" containerID="cri-o://ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37" gracePeriod=30
Sep 15 16:42:23.530678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.530596    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kube-rbac-proxy" containerID="cri-o://d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be" gracePeriod=30
Sep 15 16:42:23.625936 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.625893    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"]
Sep 15 16:42:23.628572 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.628544    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.630873 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.630852    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-da39c-predictor-serving-cert\""
Sep 15 16:42:23.630990 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.630883    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-da39c-kube-rbac-proxy-sar-config\""
Sep 15 16:42:23.641482 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.641456    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"]
Sep 15 16:42:23.716796 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.716768    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"]
Sep 15 16:42:23.719301 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.719281    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.721610 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.721582    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-da39c-predictor-serving-cert\""
Sep 15 16:42:23.721721 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.721634    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-da39c-kube-rbac-proxy-sar-config\""
Sep 15 16:42:23.731289 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.731266    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"]
Sep 15 16:42:23.739241 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.739213    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.739376 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.739257    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.739376 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.739277    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrsvm\" (UniqueName: \"kubernetes.io/projected/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-api-access-rrsvm\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.739376 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.739307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.739586 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.739380    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.840228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840193    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.840228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.840445 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.840445 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.840445 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840337    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrsvm\" (UniqueName: \"kubernetes.io/projected/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-api-access-rrsvm\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.840445 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840355    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.840445 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840380    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q597k\" (UniqueName: \"kubernetes.io/projected/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-api-access-q597k\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.840445 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840433    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98da109b-e540-46b4-b30a-03e1055b1ca3-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.840736 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98da109b-e540-46b4-b30a-03e1055b1ca3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.840736 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.840736 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.840661    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.841074 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.841052    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.841239 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.841218    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.842997 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.842973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.848027 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.847996    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrsvm\" (UniqueName: \"kubernetes.io/projected/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-api-access-rrsvm\") pod \"isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.938928 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.938838    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:23.940973 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.940948    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q597k\" (UniqueName: \"kubernetes.io/projected/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-api-access-q597k\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.941133 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.940987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98da109b-e540-46b4-b30a-03e1055b1ca3-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.941133 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.941015    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98da109b-e540-46b4-b30a-03e1055b1ca3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.941133 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.941061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.941133 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.941088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.941558 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.941533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98da109b-e540-46b4-b30a-03e1055b1ca3-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.941940 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.941915    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.942373 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.942347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.943789 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.943770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98da109b-e540-46b4-b30a-03e1055b1ca3-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:23.949428 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:23.949387    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q597k\" (UniqueName: \"kubernetes.io/projected/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-api-access-q597k\") pod \"isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:24.029954 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:24.029922    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:24.107269 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:24.107233    2564 generic.go:356] "Generic (PLEG): container finished" podID="a771202e-8252-4280-a4d6-5331082dd590" containerID="d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be" exitCode=2
Sep 15 16:42:24.107625 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:24.107305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" event={"ID":"a771202e-8252-4280-a4d6-5331082dd590","Type":"ContainerDied","Data":"d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be"}
Sep 15 16:42:24.212781 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:24.212758    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"]
Sep 15 16:42:24.214882 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:42:24.214859    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bdf5fba_52ba_4d1d_b9c5_eea906ddf3b2.slice/crio-e7d243aabc58b8d74a3a3730fa6b957d3578cf5ef60a12d2536656f0e506e8a3 WatchSource:0}: Error finding container e7d243aabc58b8d74a3a3730fa6b957d3578cf5ef60a12d2536656f0e506e8a3: Status 404 returned error can't find the container with id e7d243aabc58b8d74a3a3730fa6b957d3578cf5ef60a12d2536656f0e506e8a3
Sep 15 16:42:24.301296 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:24.301273    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"]
Sep 15 16:42:24.304059 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:42:24.304038    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98da109b_e540_46b4_b30a_03e1055b1ca3.slice/crio-17ae9d26d4096f4e2879ffd0f049b5cf30ef1be569c5ad2596de1b4d154c5e66 WatchSource:0}: Error finding container 17ae9d26d4096f4e2879ffd0f049b5cf30ef1be569c5ad2596de1b4d154c5e66: Status 404 returned error can't find the container with id 17ae9d26d4096f4e2879ffd0f049b5cf30ef1be569c5ad2596de1b4d154c5e66
Sep 15 16:42:25.047449 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:25.047380    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:42:25.113117 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:25.113081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" event={"ID":"98da109b-e540-46b4-b30a-03e1055b1ca3","Type":"ContainerStarted","Data":"e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8"}
Sep 15 16:42:25.113608 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:25.113177    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" event={"ID":"98da109b-e540-46b4-b30a-03e1055b1ca3","Type":"ContainerStarted","Data":"17ae9d26d4096f4e2879ffd0f049b5cf30ef1be569c5ad2596de1b4d154c5e66"}
Sep 15 16:42:25.114704 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:25.114682    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" event={"ID":"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2","Type":"ContainerStarted","Data":"68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625"}
Sep 15 16:42:25.114789 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:25.114727    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" event={"ID":"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2","Type":"ContainerStarted","Data":"e7d243aabc58b8d74a3a3730fa6b957d3578cf5ef60a12d2536656f0e506e8a3"}
Sep 15 16:42:26.052734 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:26.052695    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:42:27.568679 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.568656    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:42:27.673074 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.672995    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-cabundle-cert\") pod \"a771202e-8252-4280-a4d6-5331082dd590\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") "
Sep 15 16:42:27.673221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.673087    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-kube-rbac-proxy-sar-config\") pod \"a771202e-8252-4280-a4d6-5331082dd590\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") "
Sep 15 16:42:27.673221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.673141    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxvbd\" (UniqueName: \"kubernetes.io/projected/a771202e-8252-4280-a4d6-5331082dd590-kube-api-access-vxvbd\") pod \"a771202e-8252-4280-a4d6-5331082dd590\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") "
Sep 15 16:42:27.673221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.673167    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a771202e-8252-4280-a4d6-5331082dd590-kserve-provision-location\") pod \"a771202e-8252-4280-a4d6-5331082dd590\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") "
Sep 15 16:42:27.673221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.673197    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771202e-8252-4280-a4d6-5331082dd590-proxy-tls\") pod \"a771202e-8252-4280-a4d6-5331082dd590\" (UID: \"a771202e-8252-4280-a4d6-5331082dd590\") "
Sep 15 16:42:27.673484 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.673452    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a771202e-8252-4280-a4d6-5331082dd590" (UID: "a771202e-8252-4280-a4d6-5331082dd590"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:42:27.673576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.673502    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a771202e-8252-4280-a4d6-5331082dd590" (UID: "a771202e-8252-4280-a4d6-5331082dd590"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:42:27.673576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.673554    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a771202e-8252-4280-a4d6-5331082dd590-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a771202e-8252-4280-a4d6-5331082dd590" (UID: "a771202e-8252-4280-a4d6-5331082dd590"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:42:27.675316 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.675293    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a771202e-8252-4280-a4d6-5331082dd590-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a771202e-8252-4280-a4d6-5331082dd590" (UID: "a771202e-8252-4280-a4d6-5331082dd590"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:42:27.675316 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.675300    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a771202e-8252-4280-a4d6-5331082dd590-kube-api-access-vxvbd" (OuterVolumeSpecName: "kube-api-access-vxvbd") pod "a771202e-8252-4280-a4d6-5331082dd590" (UID: "a771202e-8252-4280-a4d6-5331082dd590"). InnerVolumeSpecName "kube-api-access-vxvbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:42:27.774218 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.774175    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxvbd\" (UniqueName: \"kubernetes.io/projected/a771202e-8252-4280-a4d6-5331082dd590-kube-api-access-vxvbd\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:27.774218 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.774210    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a771202e-8252-4280-a4d6-5331082dd590-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:27.774218 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.774224    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a771202e-8252-4280-a4d6-5331082dd590-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:27.774496 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.774237    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:27.774496 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:27.774249    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a771202e-8252-4280-a4d6-5331082dd590-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:28.084977 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.084932    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:42:28.129526 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.129493    2564 generic.go:356] "Generic (PLEG): container finished" podID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerID="68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625" exitCode=0
Sep 15 16:42:28.129682 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.129570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" event={"ID":"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2","Type":"ContainerDied","Data":"68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625"}
Sep 15 16:42:28.131624 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.131598    2564 generic.go:356] "Generic (PLEG): container finished" podID="a771202e-8252-4280-a4d6-5331082dd590" containerID="ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37" exitCode=0
Sep 15 16:42:28.131727 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.131656    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"
Sep 15 16:42:28.131727 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.131687    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" event={"ID":"a771202e-8252-4280-a4d6-5331082dd590","Type":"ContainerDied","Data":"ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37"}
Sep 15 16:42:28.131820 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.131746    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt" event={"ID":"a771202e-8252-4280-a4d6-5331082dd590","Type":"ContainerDied","Data":"9d91638d40f0d160f118621479dae3e5f174fdda00d5e412e433ef2b7792a524"}
Sep 15 16:42:28.131820 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.131785    2564 scope.go:117] "RemoveContainer" containerID="d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be"
Sep 15 16:42:28.143124 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.143108    2564 scope.go:117] "RemoveContainer" containerID="ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37"
Sep 15 16:42:28.153483 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.153460    2564 scope.go:117] "RemoveContainer" containerID="ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f"
Sep 15 16:42:28.162833 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.162721    2564 scope.go:117] "RemoveContainer" containerID="d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be"
Sep 15 16:42:28.163095 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:28.163070    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be\": container with ID starting with d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be not found: ID does not exist" containerID="d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be"
Sep 15 16:42:28.163202 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.163104    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be"} err="failed to get container status \"d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be\": rpc error: code = NotFound desc = could not find container \"d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be\": container with ID starting with d8fb3cc620ad71471091f7149e25be44ab22c414001219eef23de1b21c1654be not found: ID does not exist"
Sep 15 16:42:28.163202 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.163127    2564 scope.go:117] "RemoveContainer" containerID="ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37"
Sep 15 16:42:28.163540 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:28.163501    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37\": container with ID starting with ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37 not found: ID does not exist" containerID="ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37"
Sep 15 16:42:28.163617 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.163553    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37"} err="failed to get container status \"ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37\": rpc error: code = NotFound desc = could not find container \"ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37\": container with ID starting with ad5f93542c045e9529423f589e19a80268d3aad70b6268e235303a1980ad9b37 not found: ID does not exist"
Sep 15 16:42:28.163617 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.163575    2564 scope.go:117] "RemoveContainer" containerID="ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f"
Sep 15 16:42:28.164102 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:28.164068    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f\": container with ID starting with ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f not found: ID does not exist" containerID="ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f"
Sep 15 16:42:28.164197 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.164100    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f"} err="failed to get container status \"ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f\": rpc error: code = NotFound desc = could not find container \"ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f\": container with ID starting with ba02609c5773883a09a58a8cb4137066faa2f034a72209f5af06f22473a5c95f not found: ID does not exist"
Sep 15 16:42:28.165442 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.165423    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"]
Sep 15 16:42:28.165514 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:28.165454    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-e2542-predictor-6698479f49-jvnlt"]
Sep 15 16:42:29.138114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:29.138078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" event={"ID":"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2","Type":"ContainerStarted","Data":"b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6"}
Sep 15 16:42:29.138553 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:29.138174    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" event={"ID":"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2","Type":"ContainerStarted","Data":"a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4"}
Sep 15 16:42:29.138553 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:29.138416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:29.140443 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:29.140418    2564 generic.go:356] "Generic (PLEG): container finished" podID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerID="e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8" exitCode=0
Sep 15 16:42:29.140543 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:29.140463    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" event={"ID":"98da109b-e540-46b4-b30a-03e1055b1ca3","Type":"ContainerDied","Data":"e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8"}
Sep 15 16:42:29.159605 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:29.159558    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podStartSLOduration=2.344213851 podStartE2EDuration="6.159541964s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.815328113s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:42:29.157822502 +0000 UTC m=+953.493777443" watchObservedRunningTime="2026-09-15 16:42:29.159541964 +0000 UTC m=+953.495496895"
Sep 15 16:42:30.148129 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.148078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" event={"ID":"98da109b-e540-46b4-b30a-03e1055b1ca3","Type":"ContainerStarted","Data":"26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471"}
Sep 15 16:42:30.148590 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.148163    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" event={"ID":"98da109b-e540-46b4-b30a-03e1055b1ca3","Type":"ContainerStarted","Data":"24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4"}
Sep 15 16:42:30.148762 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.148743    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:30.148911 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.148883    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:30.149039 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.148929    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:30.149608 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.149583    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:42:30.149798 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.149771    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:42:30.164600 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.164572    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a771202e-8252-4280-a4d6-5331082dd590" path="/var/lib/kubelet/pods/a771202e-8252-4280-a4d6-5331082dd590/volumes"
Sep 15 16:42:30.166274 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:30.166236    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podStartSLOduration=3.267472042 podStartE2EDuration="7.166221102s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.89874906s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:42:30.165189057 +0000 UTC m=+954.501143989" watchObservedRunningTime="2026-09-15 16:42:30.166221102 +0000 UTC m=+954.502176033"
Sep 15 16:42:31.153604 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:31.153556    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:42:31.153983 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:31.153680    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:42:33.083920 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:33.083883    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:42:35.046934 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:35.046889    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:42:36.053164 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:36.053121    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:42:36.158713 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:36.158686    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:42:36.159198 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:36.159168    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:42:36.164319 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:36.164298    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:42:36.164784 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:36.164761    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:42:38.084520 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:38.084487    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:42:38.084896 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:38.084584    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:43.084008 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:43.083962    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:42:45.046918 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:45.046875    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:42:46.052808 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:46.052767    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:42:46.159319 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:46.159285    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:42:46.165228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:46.165173    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:42:48.084530 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:48.084491    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:42:53.084521 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:53.084480    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:42:54.031291 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.031268    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:54.096308 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.096273    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls\") pod \"cdb01404-179e-4f58-a3f6-662a72e23ff2\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") "
Sep 15 16:42:54.096308 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.096311    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb01404-179e-4f58-a3f6-662a72e23ff2-openshift-service-ca-bundle\") pod \"cdb01404-179e-4f58-a3f6-662a72e23ff2\" (UID: \"cdb01404-179e-4f58-a3f6-662a72e23ff2\") "
Sep 15 16:42:54.096772 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.096669    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdb01404-179e-4f58-a3f6-662a72e23ff2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cdb01404-179e-4f58-a3f6-662a72e23ff2" (UID: "cdb01404-179e-4f58-a3f6-662a72e23ff2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:42:54.098600 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.098582    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cdb01404-179e-4f58-a3f6-662a72e23ff2" (UID: "cdb01404-179e-4f58-a3f6-662a72e23ff2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:42:54.197821 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.197791    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdb01404-179e-4f58-a3f6-662a72e23ff2-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:54.197821 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.197815    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cdb01404-179e-4f58-a3f6-662a72e23ff2-openshift-service-ca-bundle\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:42:54.250906 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.250869    2564 generic.go:356] "Generic (PLEG): container finished" podID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerID="dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b" exitCode=0
Sep 15 16:42:54.251097 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.250932    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"
Sep 15 16:42:54.251097 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.250968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" event={"ID":"cdb01404-179e-4f58-a3f6-662a72e23ff2","Type":"ContainerDied","Data":"dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b"}
Sep 15 16:42:54.251097 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.251032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d" event={"ID":"cdb01404-179e-4f58-a3f6-662a72e23ff2","Type":"ContainerDied","Data":"c38938f63ffcabc138f4b149e48c31c41b96f450e135282038065c4cec954317"}
Sep 15 16:42:54.251097 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.251065    2564 scope.go:117] "RemoveContainer" containerID="dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b"
Sep 15 16:42:54.259222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.259205    2564 scope.go:117] "RemoveContainer" containerID="dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b"
Sep 15 16:42:54.259563 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:42:54.259543    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b\": container with ID starting with dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b not found: ID does not exist" containerID="dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b"
Sep 15 16:42:54.259618 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.259572    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b"} err="failed to get container status \"dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b\": rpc error: code = NotFound desc = could not find container \"dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b\": container with ID starting with dbd10d9c4fc5d79be7b5f59bad641870c612c20d74915f486a9708ae448b312b not found: ID does not exist"
Sep 15 16:42:54.266077 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.266051    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"]
Sep 15 16:42:54.267788 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:54.267768    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-e2542-7fdc9d5f94-mpc8d"]
Sep 15 16:42:55.046824 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:55.046782    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:42:56.053562 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:56.053518    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:42:56.159671 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:56.159631    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:42:56.164547 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:56.164521    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" path="/var/lib/kubelet/pods/cdb01404-179e-4f58-a3f6-662a72e23ff2/volumes"
Sep 15 16:42:56.164816 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:42:56.164793    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:43:05.047020 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:05.046806    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:43:06.052843 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:06.052807    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:43:06.159285 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:06.159246    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:43:06.164884 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:06.164848    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:43:15.047473 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:15.047426    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:43:16.053564 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:16.053526    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:43:16.159646 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:16.159611    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:43:16.167491 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:16.165008    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:43:20.160388 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:20.160342    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Sep 15 16:43:26.053664 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:26.053623    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:43:26.159110 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:26.159073    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:43:26.165262 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:26.165235    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:43:30.164633 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:30.164597    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:43:30.280951 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:30.280918    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"]
Sep 15 16:43:30.281166 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:30.281144    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" containerID="cri-o://2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69" gracePeriod=30
Sep 15 16:43:30.281254 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:30.281214    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kube-rbac-proxy" containerID="cri-o://01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8" gracePeriod=30
Sep 15 16:43:30.281471 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:30.281434    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 15 16:43:30.405502 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:30.405474    2564 generic.go:356] "Generic (PLEG): container finished" podID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerID="01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8" exitCode=2
Sep 15 16:43:30.405628 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:30.405516    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" event={"ID":"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2","Type":"ContainerDied","Data":"01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8"}
Sep 15 16:43:31.047031 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:31.046990    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 15 16:43:33.426380 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:33.426284    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"]
Sep 15 16:43:33.426856 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:33.426542    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kube-rbac-proxy" containerID="cri-o://e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9" gracePeriod=30
Sep 15 16:43:33.426856 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:33.426518    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container" containerID="cri-o://0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061" gracePeriod=30
Sep 15 16:43:34.423056 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:34.423026    2564 generic.go:356] "Generic (PLEG): container finished" podID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerID="e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9" exitCode=2
Sep 15 16:43:34.423216 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:34.423097    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" event={"ID":"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9","Type":"ContainerDied","Data":"e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9"}
Sep 15 16:43:35.042036 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:35.041994    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Sep 15 16:43:36.046873 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.046807    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 15 16:43:36.130841 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.130813    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:43:36.159976 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.159938    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:43:36.165167 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.165141    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:43:36.270118 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270083    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-rbac-proxy-sar-config\") pod \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") "
Sep 15 16:43:36.270294 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270189    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls\") pod \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") "
Sep 15 16:43:36.270294 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270248    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kserve-provision-location\") pod \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") "
Sep 15 16:43:36.270294 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270289    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj89g\" (UniqueName: \"kubernetes.io/projected/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-api-access-rj89g\") pod \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") "
Sep 15 16:43:36.270488 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270322    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-cabundle-cert\") pod \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\" (UID: \"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2\") "
Sep 15 16:43:36.270488 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270469    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" (UID: "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:43:36.270654 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270625    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" (UID: "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:43:36.270715 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270639    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:36.270759 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.270709    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" (UID: "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:43:36.272507 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.272490    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" (UID: "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:43:36.272583 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.272566    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-api-access-rj89g" (OuterVolumeSpecName: "kube-api-access-rj89g") pod "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" (UID: "51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2"). InnerVolumeSpecName "kube-api-access-rj89g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:43:36.371116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.371088    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:36.371116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.371112    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj89g\" (UniqueName: \"kubernetes.io/projected/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-kube-api-access-rj89g\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:36.371116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.371122    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:36.371332 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.371132    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:36.433053 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.433023    2564 generic.go:356] "Generic (PLEG): container finished" podID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerID="2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69" exitCode=0
Sep 15 16:43:36.433228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.433082    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"
Sep 15 16:43:36.433228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.433114    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" event={"ID":"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2","Type":"ContainerDied","Data":"2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69"}
Sep 15 16:43:36.433228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.433188    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx" event={"ID":"51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2","Type":"ContainerDied","Data":"572e971fb80ae368af7a10bc0e2aead59b5b7f5fd5751c09b28629b74796a885"}
Sep 15 16:43:36.433228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.433223    2564 scope.go:117] "RemoveContainer" containerID="01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8"
Sep 15 16:43:36.441980 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.441962    2564 scope.go:117] "RemoveContainer" containerID="2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69"
Sep 15 16:43:36.451054 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.450819    2564 scope.go:117] "RemoveContainer" containerID="97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d"
Sep 15 16:43:36.458294 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.458198    2564 scope.go:117] "RemoveContainer" containerID="01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8"
Sep 15 16:43:36.458537 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:43:36.458508    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8\": container with ID starting with 01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8 not found: ID does not exist" containerID="01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8"
Sep 15 16:43:36.458625 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.458542    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8"} err="failed to get container status \"01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8\": rpc error: code = NotFound desc = could not find container \"01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8\": container with ID starting with 01de4aebe1743d66794ded2fcb101aa787d63b632d997148b03513ef7dcc88e8 not found: ID does not exist"
Sep 15 16:43:36.458625 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.458560    2564 scope.go:117] "RemoveContainer" containerID="2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69"
Sep 15 16:43:36.458821 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:43:36.458803    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69\": container with ID starting with 2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69 not found: ID does not exist" containerID="2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69"
Sep 15 16:43:36.458864 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.458827    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69"} err="failed to get container status \"2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69\": rpc error: code = NotFound desc = could not find container \"2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69\": container with ID starting with 2737993de8703f87194f3d2c0abc8d3b7e830d0ab6dc0f4f79ff53ba99709f69 not found: ID does not exist"
Sep 15 16:43:36.458864 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.458845    2564 scope.go:117] "RemoveContainer" containerID="97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d"
Sep 15 16:43:36.459093 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:43:36.459077    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d\": container with ID starting with 97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d not found: ID does not exist" containerID="97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d"
Sep 15 16:43:36.459148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.459097    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d"} err="failed to get container status \"97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d\": rpc error: code = NotFound desc = could not find container \"97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d\": container with ID starting with 97bec302bf8bf8e7efd9a9a9a5533272e0b922e43a4632ba7b5a7ce203e7641d not found: ID does not exist"
Sep 15 16:43:36.466140 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.466115    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"]
Sep 15 16:43:36.475955 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:36.475935    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-4bzqx"]
Sep 15 16:43:37.160066 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:37.160022    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:43:38.164871 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.164847    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" path="/var/lib/kubelet/pods/51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2/volumes"
Sep 15 16:43:38.377521 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.377498    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:43:38.389294 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.389277    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-cabundle-cert\") pod \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") "
Sep 15 16:43:38.389362 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.389317    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-proxy-tls\") pod \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") "
Sep 15 16:43:38.389416 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.389368    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-rbac-proxy-sar-config\") pod \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") "
Sep 15 16:43:38.389416 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.389386    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kserve-provision-location\") pod \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") "
Sep 15 16:43:38.389498 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.389435    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvgqh\" (UniqueName: \"kubernetes.io/projected/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-api-access-lvgqh\") pod \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\" (UID: \"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9\") "
Sep 15 16:43:38.389707 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.389688    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" (UID: "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:43:38.389808 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.389733    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" (UID: "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:43:38.389808 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.389752    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" (UID: "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:43:38.391561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.391539    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-api-access-lvgqh" (OuterVolumeSpecName: "kube-api-access-lvgqh") pod "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" (UID: "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9"). InnerVolumeSpecName "kube-api-access-lvgqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:43:38.391621 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.391605    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" (UID: "50b3dd63-751d-494b-b7b7-b6ef0f6a63f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:43:38.444453 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.444352    2564 generic.go:356] "Generic (PLEG): container finished" podID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerID="0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061" exitCode=0
Sep 15 16:43:38.444453 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.444432    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"
Sep 15 16:43:38.444453 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.444430    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" event={"ID":"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9","Type":"ContainerDied","Data":"0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061"}
Sep 15 16:43:38.444670 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.444513    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k" event={"ID":"50b3dd63-751d-494b-b7b7-b6ef0f6a63f9","Type":"ContainerDied","Data":"556cf8a5874bef3f7985a028b668dd94893a6a1129cffd15dcd1c8d135223cea"}
Sep 15 16:43:38.444670 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.444553    2564 scope.go:117] "RemoveContainer" containerID="e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9"
Sep 15 16:43:38.453517 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.453492    2564 scope.go:117] "RemoveContainer" containerID="0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061"
Sep 15 16:43:38.461652 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.461632    2564 scope.go:117] "RemoveContainer" containerID="66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf"
Sep 15 16:43:38.466039 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.466019    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"]
Sep 15 16:43:38.469249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.469233    2564 scope.go:117] "RemoveContainer" containerID="e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9"
Sep 15 16:43:38.469909 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:43:38.469877    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9\": container with ID starting with e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9 not found: ID does not exist" containerID="e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9"
Sep 15 16:43:38.470019 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.469920    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9"} err="failed to get container status \"e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9\": rpc error: code = NotFound desc = could not find container \"e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9\": container with ID starting with e6cad7979aee34bfeda0fdb0deda9d10e8f286d3edb2d9269b0157802b1e6cf9 not found: ID does not exist"
Sep 15 16:43:38.470019 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.469941    2564 scope.go:117] "RemoveContainer" containerID="0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061"
Sep 15 16:43:38.470282 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:43:38.470262    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061\": container with ID starting with 0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061 not found: ID does not exist" containerID="0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061"
Sep 15 16:43:38.470341 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.470289    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061"} err="failed to get container status \"0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061\": rpc error: code = NotFound desc = could not find container \"0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061\": container with ID starting with 0710736d95423b61cd6ae984658a2bc8936339aaf34dd25b44a219c733763061 not found: ID does not exist"
Sep 15 16:43:38.470341 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.470310    2564 scope.go:117] "RemoveContainer" containerID="66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf"
Sep 15 16:43:38.470697 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:43:38.470673    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf\": container with ID starting with 66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf not found: ID does not exist" containerID="66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf"
Sep 15 16:43:38.470772 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.470707    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf"} err="failed to get container status \"66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf\": rpc error: code = NotFound desc = could not find container \"66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf\": container with ID starting with 66ccca48d9d68b12e0d9329231daeefe86b56f81fa0615122b51e596d4270adf not found: ID does not exist"
Sep 15 16:43:38.471784 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.471767    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-jh56k"]
Sep 15 16:43:38.490279 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.490253    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvgqh\" (UniqueName: \"kubernetes.io/projected/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-api-access-lvgqh\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:38.490279 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.490275    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:38.490540 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.490285    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:38.490540 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.490297    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:38.490540 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:38.490308    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:43:40.163828 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:40.163797    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" path="/var/lib/kubelet/pods/50b3dd63-751d-494b-b7b7-b6ef0f6a63f9/volumes"
Sep 15 16:43:43.697248 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697219    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"]
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697579    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697594    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697612    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="storage-initializer"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697617    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="storage-initializer"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697624    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="storage-initializer"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697629    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="storage-initializer"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697637    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697642    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697647    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697653    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697660    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="storage-initializer"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697665    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="storage-initializer"
Sep 15 16:43:43.697678 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697681    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697689    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697697    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697701    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697708    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697714    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697722    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697727    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697781    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697789    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697796    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a771202e-8252-4280-a4d6-5331082dd590" containerName="kserve-container"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697802    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kserve-container"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697808    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="50b3dd63-751d-494b-b7b7-b6ef0f6a63f9" containerName="kube-rbac-proxy"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697813    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="51aba90f-0bfa-49c2-ae3b-1fb7d3ea5bd2" containerName="kserve-container"
Sep 15 16:43:43.698085 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.697818    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdb01404-179e-4f58-a3f6-662a72e23ff2" containerName="model-chainer-raw-e2542"
Sep 15 16:43:43.702731 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.702717    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.704963 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.704946    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-97930a-predictor-serving-cert\""
Sep 15 16:43:43.704963 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.704956    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-97930a-kube-rbac-proxy-sar-config\""
Sep 15 16:43:43.710033 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.710010    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"]
Sep 15 16:43:43.733690 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.733671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kserve-provision-location\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.733814 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.733704    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.733814 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.733728    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.733814 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.733761    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wmx4\" (UniqueName: \"kubernetes.io/projected/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-api-access-4wmx4\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.733953 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.733823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-cabundle-cert\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.834815 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.834785    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-cabundle-cert\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.835008 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.834845    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kserve-provision-location\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.835008 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.834872    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.835008 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.834892    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.835008 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.834912    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wmx4\" (UniqueName: \"kubernetes.io/projected/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-api-access-4wmx4\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.835238 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:43:43.835021    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-97930a-predictor-serving-cert: secret "isvc-primary-97930a-predictor-serving-cert" not found
Sep 15 16:43:43.835238 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:43:43.835111    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls podName:432d4def-10d4-4ca9-9b8a-d6b48d6134f9 nodeName:}" failed. No retries permitted until 2026-09-15 16:43:44.335095158 +0000 UTC m=+1028.671050067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls") pod "isvc-primary-97930a-predictor-75d84c5688-fb25c" (UID: "432d4def-10d4-4ca9-9b8a-d6b48d6134f9") : secret "isvc-primary-97930a-predictor-serving-cert" not found
Sep 15 16:43:43.835356 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.835290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kserve-provision-location\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.835584 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.835565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-rbac-proxy-sar-config\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.835652 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.835581    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-cabundle-cert\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:43.843373 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:43.843348    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wmx4\" (UniqueName: \"kubernetes.io/projected/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-api-access-4wmx4\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:44.339468 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:44.339432    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:44.342152 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:44.342122    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls\") pod \"isvc-primary-97930a-predictor-75d84c5688-fb25c\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") " pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:44.613950 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:44.613849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:44.886656 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:44.886624    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"]
Sep 15 16:43:44.889096 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:43:44.889070    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432d4def_10d4_4ca9_9b8a_d6b48d6134f9.slice/crio-530da0371fa4b75c946624d438f523d05a50dfa03db644808fa95b06b51bca51 WatchSource:0}: Error finding container 530da0371fa4b75c946624d438f523d05a50dfa03db644808fa95b06b51bca51: Status 404 returned error can't find the container with id 530da0371fa4b75c946624d438f523d05a50dfa03db644808fa95b06b51bca51
Sep 15 16:43:45.482479 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:45.482443    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" event={"ID":"432d4def-10d4-4ca9-9b8a-d6b48d6134f9","Type":"ContainerStarted","Data":"2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210"}
Sep 15 16:43:45.482653 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:45.482524    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" event={"ID":"432d4def-10d4-4ca9-9b8a-d6b48d6134f9","Type":"ContainerStarted","Data":"530da0371fa4b75c946624d438f523d05a50dfa03db644808fa95b06b51bca51"}
Sep 15 16:43:46.165531 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:46.165506    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:43:47.160546 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:47.160509    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:43:49.500364 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:49.500335    2564 generic.go:356] "Generic (PLEG): container finished" podID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerID="2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210" exitCode=0
Sep 15 16:43:49.500762 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:49.500434    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" event={"ID":"432d4def-10d4-4ca9-9b8a-d6b48d6134f9","Type":"ContainerDied","Data":"2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210"}
Sep 15 16:43:50.506538 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:50.506505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" event={"ID":"432d4def-10d4-4ca9-9b8a-d6b48d6134f9","Type":"ContainerStarted","Data":"46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046"}
Sep 15 16:43:50.506918 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:50.506571    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" event={"ID":"432d4def-10d4-4ca9-9b8a-d6b48d6134f9","Type":"ContainerStarted","Data":"6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5"}
Sep 15 16:43:50.506918 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:50.506834    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:50.507028 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:50.506994    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:50.508408 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:50.508369    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:43:50.524809 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:50.524765    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podStartSLOduration=3.816948666 podStartE2EDuration="7.524753778s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.707805112s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:43:50.523421007 +0000 UTC m=+1034.859375936" watchObservedRunningTime="2026-09-15 16:43:50.524753778 +0000 UTC m=+1034.860708709"
Sep 15 16:43:51.511236 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:51.511192    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:43:56.516054 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:56.516023    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:43:56.516571 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:56.516538    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:43:57.160576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:43:57.160549    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:44:06.517307 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:06.517259    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:44:13.618925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.618894    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"]
Sep 15 16:44:13.621985 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.621970    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:13.624226 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.624206    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-da39c-kube-rbac-proxy-sar-config\""
Sep 15 16:44:13.624362 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.624238    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-da39c-serving-cert\""
Sep 15 16:44:13.629822 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.629799    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"]
Sep 15 16:44:13.682455 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.682417    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8d2de2-37f2-4e49-93a2-0580570e87c2-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:13.682625 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.682467    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls\") pod \"model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:13.783645 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.783611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8d2de2-37f2-4e49-93a2-0580570e87c2-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:13.783645 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.783651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls\") pod \"model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:13.783867 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:13.783798    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-serving-cert: secret "model-chainer-raw-hpa-da39c-serving-cert" not found
Sep 15 16:44:13.783907 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:13.783872    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls podName:dc8d2de2-37f2-4e49-93a2-0580570e87c2 nodeName:}" failed. No retries permitted until 2026-09-15 16:44:14.283855645 +0000 UTC m=+1058.619810554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls") pod "model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" (UID: "dc8d2de2-37f2-4e49-93a2-0580570e87c2") : secret "model-chainer-raw-hpa-da39c-serving-cert" not found
Sep 15 16:44:13.784248 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:13.784229    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8d2de2-37f2-4e49-93a2-0580570e87c2-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:14.288065 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:14.288027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls\") pod \"model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:14.290626 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:14.290597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls\") pod \"model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:14.532664 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:14.532630    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:14.801672 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:14.801647    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"]
Sep 15 16:44:14.804159 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:44:14.804129    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8d2de2_37f2_4e49_93a2_0580570e87c2.slice/crio-23954f7198c2a853391bcb4666b1238f459447c4dee01490982fac8fca35a9b4 WatchSource:0}: Error finding container 23954f7198c2a853391bcb4666b1238f459447c4dee01490982fac8fca35a9b4: Status 404 returned error can't find the container with id 23954f7198c2a853391bcb4666b1238f459447c4dee01490982fac8fca35a9b4
Sep 15 16:44:15.616656 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:15.616617    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" event={"ID":"dc8d2de2-37f2-4e49-93a2-0580570e87c2","Type":"ContainerStarted","Data":"43f6f4c43e2b7f876bc896fdd90d362ce27cbeee0a5492180c5e2a6d77810366"}
Sep 15 16:44:15.616822 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:15.616680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" event={"ID":"dc8d2de2-37f2-4e49-93a2-0580570e87c2","Type":"ContainerStarted","Data":"23954f7198c2a853391bcb4666b1238f459447c4dee01490982fac8fca35a9b4"}
Sep 15 16:44:15.616822 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:15.616710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:15.631903 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:15.631856    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" podStartSLOduration=2.631840198 podStartE2EDuration="2.631840198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:44:15.630860237 +0000 UTC m=+1059.966815158" watchObservedRunningTime="2026-09-15 16:44:15.631840198 +0000 UTC m=+1059.967795130"
Sep 15 16:44:16.517253 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:16.517217    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:44:21.626104 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:21.626076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:23.677430 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:23.677376    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"]
Sep 15 16:44:23.677826 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:23.677603    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c" containerID="cri-o://43f6f4c43e2b7f876bc896fdd90d362ce27cbeee0a5492180c5e2a6d77810366" gracePeriod=30
Sep 15 16:44:23.761836 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:23.761802    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"]
Sep 15 16:44:23.762136 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:23.762108    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" containerID="cri-o://a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4" gracePeriod=30
Sep 15 16:44:23.762202 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:23.762178    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kube-rbac-proxy" containerID="cri-o://b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6" gracePeriod=30
Sep 15 16:44:23.808840 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:23.808810    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"]
Sep 15 16:44:23.809082 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:23.809042    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" containerID="cri-o://24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4" gracePeriod=30
Sep 15 16:44:23.809163 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:23.809062    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kube-rbac-proxy" containerID="cri-o://26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471" gracePeriod=30
Sep 15 16:44:24.658924 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:24.658893    2564 generic.go:356] "Generic (PLEG): container finished" podID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerID="b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6" exitCode=2
Sep 15 16:44:24.659119 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:24.658969    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" event={"ID":"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2","Type":"ContainerDied","Data":"b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6"}
Sep 15 16:44:24.661139 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:24.661121    2564 generic.go:356] "Generic (PLEG): container finished" podID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerID="26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471" exitCode=2
Sep 15 16:44:24.661249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:24.661157    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" event={"ID":"98da109b-e540-46b4-b30a-03e1055b1ca3","Type":"ContainerDied","Data":"26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471"}
Sep 15 16:44:26.154168 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:26.154125    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 15 16:44:26.154571 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:26.154138    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 15 16:44:26.165448 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:26.165419    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 15 16:44:26.517313 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:26.517280    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:44:26.624786 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:26.624739    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:27.160559 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:27.160519    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 15 16:44:28.156902 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.156875    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:44:28.303357 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303324    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98da109b-e540-46b4-b30a-03e1055b1ca3-proxy-tls\") pod \"98da109b-e540-46b4-b30a-03e1055b1ca3\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") "
Sep 15 16:44:28.303800 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303372    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98da109b-e540-46b4-b30a-03e1055b1ca3-kserve-provision-location\") pod \"98da109b-e540-46b4-b30a-03e1055b1ca3\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") "
Sep 15 16:44:28.303800 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303432    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-cabundle-cert\") pod \"98da109b-e540-46b4-b30a-03e1055b1ca3\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") "
Sep 15 16:44:28.303800 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303523    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q597k\" (UniqueName: \"kubernetes.io/projected/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-api-access-q597k\") pod \"98da109b-e540-46b4-b30a-03e1055b1ca3\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") "
Sep 15 16:44:28.303800 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303621    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-rbac-proxy-sar-config\") pod \"98da109b-e540-46b4-b30a-03e1055b1ca3\" (UID: \"98da109b-e540-46b4-b30a-03e1055b1ca3\") "
Sep 15 16:44:28.303800 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303706    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98da109b-e540-46b4-b30a-03e1055b1ca3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "98da109b-e540-46b4-b30a-03e1055b1ca3" (UID: "98da109b-e540-46b4-b30a-03e1055b1ca3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:44:28.303800 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303757    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "98da109b-e540-46b4-b30a-03e1055b1ca3" (UID: "98da109b-e540-46b4-b30a-03e1055b1ca3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:28.304063 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303968    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/98da109b-e540-46b4-b30a-03e1055b1ca3-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.304063 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303989    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.304063 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.303995    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98da109b-e540-46b4-b30a-03e1055b1ca3" (UID: "98da109b-e540-46b4-b30a-03e1055b1ca3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:28.305544 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.305522    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98da109b-e540-46b4-b30a-03e1055b1ca3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98da109b-e540-46b4-b30a-03e1055b1ca3" (UID: "98da109b-e540-46b4-b30a-03e1055b1ca3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:44:28.305674 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.305651    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-api-access-q597k" (OuterVolumeSpecName: "kube-api-access-q597k") pod "98da109b-e540-46b4-b30a-03e1055b1ca3" (UID: "98da109b-e540-46b4-b30a-03e1055b1ca3"). InnerVolumeSpecName "kube-api-access-q597k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:44:28.404776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.404742    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q597k\" (UniqueName: \"kubernetes.io/projected/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-api-access-q597k\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.404776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.404772    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98da109b-e540-46b4-b30a-03e1055b1ca3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.404776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.404781    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98da109b-e540-46b4-b30a-03e1055b1ca3-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.600878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.600858    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:44:28.682771 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.682733    2564 generic.go:356] "Generic (PLEG): container finished" podID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerID="24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4" exitCode=0
Sep 15 16:44:28.682941 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.682793    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"
Sep 15 16:44:28.682941 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.682812    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" event={"ID":"98da109b-e540-46b4-b30a-03e1055b1ca3","Type":"ContainerDied","Data":"24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4"}
Sep 15 16:44:28.682941 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.682872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd" event={"ID":"98da109b-e540-46b4-b30a-03e1055b1ca3","Type":"ContainerDied","Data":"17ae9d26d4096f4e2879ffd0f049b5cf30ef1be569c5ad2596de1b4d154c5e66"}
Sep 15 16:44:28.682941 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.682887    2564 scope.go:117] "RemoveContainer" containerID="26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471"
Sep 15 16:44:28.685081 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.685060    2564 generic.go:356] "Generic (PLEG): container finished" podID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerID="a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4" exitCode=0
Sep 15 16:44:28.685208 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.685088    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" event={"ID":"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2","Type":"ContainerDied","Data":"a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4"}
Sep 15 16:44:28.685208 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.685118    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"
Sep 15 16:44:28.685208 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.685129    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9" event={"ID":"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2","Type":"ContainerDied","Data":"e7d243aabc58b8d74a3a3730fa6b957d3578cf5ef60a12d2536656f0e506e8a3"}
Sep 15 16:44:28.691449 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.691436    2564 scope.go:117] "RemoveContainer" containerID="24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4"
Sep 15 16:44:28.698764 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.698749    2564 scope.go:117] "RemoveContainer" containerID="e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8"
Sep 15 16:44:28.703115 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.703095    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"]
Sep 15 16:44:28.706511 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.706489    2564 scope.go:117] "RemoveContainer" containerID="26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471"
Sep 15 16:44:28.706778 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:28.706761    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471\": container with ID starting with 26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471 not found: ID does not exist" containerID="26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471"
Sep 15 16:44:28.706839 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.706787    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471"} err="failed to get container status \"26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471\": rpc error: code = NotFound desc = could not find container \"26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471\": container with ID starting with 26304d7dd01af039bd43dd3369c48e0c315a2df0941411119eb9eb30a2d5c471 not found: ID does not exist"
Sep 15 16:44:28.706839 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.706802    2564 scope.go:117] "RemoveContainer" containerID="24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4"
Sep 15 16:44:28.706839 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.706829    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-cabundle-cert\") pod \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") "
Sep 15 16:44:28.707000 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.706856    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-proxy-tls\") pod \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") "
Sep 15 16:44:28.707000 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.706953    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-rbac-proxy-sar-config\") pod \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") "
Sep 15 16:44:28.707102 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707010    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrsvm\" (UniqueName: \"kubernetes.io/projected/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-api-access-rrsvm\") pod \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") "
Sep 15 16:44:28.707102 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707045    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kserve-provision-location\") pod \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\" (UID: \"7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2\") "
Sep 15 16:44:28.707223 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707200    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" (UID: "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:28.707328 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707306    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.707472 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707448    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" (UID: "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:28.707593 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:28.707382    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4\": container with ID starting with 24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4 not found: ID does not exist" containerID="24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4"
Sep 15 16:44:28.707593 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707565    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4"} err="failed to get container status \"24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4\": rpc error: code = NotFound desc = could not find container \"24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4\": container with ID starting with 24b2ef3a45694739ad24646d4d6b6052851db8b0cf1f8ff4bc9afd5c05e88ad4 not found: ID does not exist"
Sep 15 16:44:28.707593 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707583    2564 scope.go:117] "RemoveContainer" containerID="e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8"
Sep 15 16:44:28.707759 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707734    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" (UID: "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:44:28.707759 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707747    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-da39c-predictor-6ff999c8d5-mfknd"]
Sep 15 16:44:28.707901 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:28.707885    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8\": container with ID starting with e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8 not found: ID does not exist" containerID="e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8"
Sep 15 16:44:28.707957 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707909    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8"} err="failed to get container status \"e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8\": rpc error: code = NotFound desc = could not find container \"e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8\": container with ID starting with e0f72c7dbc70bc69c0ca8f16f3b2976cd42c863cf3832b3abceeeba9c80f1ab8 not found: ID does not exist"
Sep 15 16:44:28.707957 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.707931    2564 scope.go:117] "RemoveContainer" containerID="b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6"
Sep 15 16:44:28.709142 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.709120    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" (UID: "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:44:28.709216 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.709183    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-api-access-rrsvm" (OuterVolumeSpecName: "kube-api-access-rrsvm") pod "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" (UID: "7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2"). InnerVolumeSpecName "kube-api-access-rrsvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:44:28.715088 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.715069    2564 scope.go:117] "RemoveContainer" containerID="a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4"
Sep 15 16:44:28.722082 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.722062    2564 scope.go:117] "RemoveContainer" containerID="68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625"
Sep 15 16:44:28.729000 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.728956    2564 scope.go:117] "RemoveContainer" containerID="b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6"
Sep 15 16:44:28.729270 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:28.729252    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6\": container with ID starting with b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6 not found: ID does not exist" containerID="b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6"
Sep 15 16:44:28.729316 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.729278    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6"} err="failed to get container status \"b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6\": rpc error: code = NotFound desc = could not find container \"b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6\": container with ID starting with b7ed37fd4f6335a735c4edb3f1cb0965b4f74ceb8071721d12999d31aca158f6 not found: ID does not exist"
Sep 15 16:44:28.729316 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.729299    2564 scope.go:117] "RemoveContainer" containerID="a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4"
Sep 15 16:44:28.729568 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:28.729536    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4\": container with ID starting with a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4 not found: ID does not exist" containerID="a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4"
Sep 15 16:44:28.729616 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.729573    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4"} err="failed to get container status \"a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4\": rpc error: code = NotFound desc = could not find container \"a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4\": container with ID starting with a80b5bb7c3c6d2d95da3858b27f4373e10639e9858ec20802c52d443d63192b4 not found: ID does not exist"
Sep 15 16:44:28.729616 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.729587    2564 scope.go:117] "RemoveContainer" containerID="68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625"
Sep 15 16:44:28.729812 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:28.729798    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625\": container with ID starting with 68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625 not found: ID does not exist" containerID="68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625"
Sep 15 16:44:28.729854 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.729816    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625"} err="failed to get container status \"68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625\": rpc error: code = NotFound desc = could not find container \"68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625\": container with ID starting with 68f8dca570ca3def60025a70dcd6f40cdfed4bc7997b74769a4df9359d3a3625 not found: ID does not exist"
Sep 15 16:44:28.808481 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.808448    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrsvm\" (UniqueName: \"kubernetes.io/projected/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-api-access-rrsvm\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.808481 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.808476    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.808481 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.808486    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:28.808693 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:28.808495    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:29.005541 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:29.005513    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"]
Sep 15 16:44:29.009159 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:29.009133    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-da39c-predictor-7bf98cc5b9-dvdr9"]
Sep 15 16:44:30.163920 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:30.163877    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" path="/var/lib/kubelet/pods/7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2/volumes"
Sep 15 16:44:30.164563 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:30.164544    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" path="/var/lib/kubelet/pods/98da109b-e540-46b4-b30a-03e1055b1ca3/volumes"
Sep 15 16:44:31.625102 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:31.625064    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:33.920738 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.920702    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"]
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921001    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="storage-initializer"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921012    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="storage-initializer"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921029    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kube-rbac-proxy"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921035    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kube-rbac-proxy"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921042    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921047    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921057    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921062    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921071    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="storage-initializer"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921075    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="storage-initializer"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921080    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kube-rbac-proxy"
Sep 15 16:44:33.921098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921085    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kube-rbac-proxy"
Sep 15 16:44:33.921499 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921128    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kserve-container"
Sep 15 16:44:33.921499 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921135    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kserve-container"
Sep 15 16:44:33.921499 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921142    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bdf5fba-52ba-4d1d-b9c5-eea906ddf3b2" containerName="kube-rbac-proxy"
Sep 15 16:44:33.921499 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.921149    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="98da109b-e540-46b4-b30a-03e1055b1ca3" containerName="kube-rbac-proxy"
Sep 15 16:44:33.924419 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.924385    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:33.926717 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.926697    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-ae4ee-predictor-serving-cert\""
Sep 15 16:44:33.926921 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.926907    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-ae4ee-kube-rbac-proxy-sar-config\""
Sep 15 16:44:33.935027 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:33.935005    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"]
Sep 15 16:44:34.051870 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.051837    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptf57\" (UniqueName: \"kubernetes.io/projected/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-api-access-ptf57\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.052064 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.051879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-cabundle-cert\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.052064 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.051918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.052064 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.051992    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kserve-provision-location\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.052064 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.052024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.153368 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.153336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-cabundle-cert\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.153368 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.153373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.153619 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.153451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kserve-provision-location\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.153619 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.153473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.153619 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.153507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptf57\" (UniqueName: \"kubernetes.io/projected/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-api-access-ptf57\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.153619 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:34.153602    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-serving-cert: secret "isvc-logger-raw-ae4ee-predictor-serving-cert" not found
Sep 15 16:44:34.153820 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:44:34.153684    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls podName:a5d193ac-709a-4fe3-a07a-202712e0d8b9 nodeName:}" failed. No retries permitted until 2026-09-15 16:44:34.653663563 +0000 UTC m=+1078.989618487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls") pod "isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" (UID: "a5d193ac-709a-4fe3-a07a-202712e0d8b9") : secret "isvc-logger-raw-ae4ee-predictor-serving-cert" not found
Sep 15 16:44:34.153876 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.153858    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kserve-provision-location\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.154010 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.153992    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-cabundle-cert\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.154043 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.154015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.164451 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.164416    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptf57\" (UniqueName: \"kubernetes.io/projected/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-api-access-ptf57\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.657578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.657538    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.660091 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.660062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls\") pod \"isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") " pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:34.834477 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:34.834443    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:35.105092 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:35.105059    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"]
Sep 15 16:44:35.108368 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:44:35.108339    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d193ac_709a_4fe3_a07a_202712e0d8b9.slice/crio-af38d4783fc2a8234cd6b411dd3ebbd314ff63adf1f2ba89dc48871016002ca3 WatchSource:0}: Error finding container af38d4783fc2a8234cd6b411dd3ebbd314ff63adf1f2ba89dc48871016002ca3: Status 404 returned error can't find the container with id af38d4783fc2a8234cd6b411dd3ebbd314ff63adf1f2ba89dc48871016002ca3
Sep 15 16:44:35.721472 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:35.721437    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerStarted","Data":"896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b"}
Sep 15 16:44:35.721648 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:35.721517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerStarted","Data":"af38d4783fc2a8234cd6b411dd3ebbd314ff63adf1f2ba89dc48871016002ca3"}
Sep 15 16:44:36.516948 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:36.516908    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:44:36.625209 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:36.625171    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:36.625421 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:36.625294    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:39.739005 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:39.738926    2564 generic.go:356] "Generic (PLEG): container finished" podID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerID="896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b" exitCode=0
Sep 15 16:44:39.739418 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:39.738999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerDied","Data":"896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b"}
Sep 15 16:44:40.745336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:40.745298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerStarted","Data":"699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8"}
Sep 15 16:44:40.745774 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:40.745384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerStarted","Data":"23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950"}
Sep 15 16:44:40.745774 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:40.745451    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerStarted","Data":"2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e"}
Sep 15 16:44:40.745774 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:40.745687    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:40.766799 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:40.766748    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podStartSLOduration=3.5911865560000003 podStartE2EDuration="7.766730972s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.175544416s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:44:40.764963558 +0000 UTC m=+1085.100918501" watchObservedRunningTime="2026-09-15 16:44:40.766730972 +0000 UTC m=+1085.102685901"
Sep 15 16:44:41.624608 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:41.624572    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:41.755046 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:41.752442    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:41.755046 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:41.752558    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:41.755046 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:41.754061    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:44:41.755046 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:41.754768    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:42.755074 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:42.755026    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:44:42.755506 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:42.755429    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:46.516462 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:46.516426    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:44:46.624914 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:46.624871    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:47.759303 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:47.759271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:44:47.759905 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:47.759877    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:44:47.760070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:47.760046    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:51.625290 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:51.625248    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:44:53.800094 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:53.800064    2564 generic.go:356] "Generic (PLEG): container finished" podID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerID="43f6f4c43e2b7f876bc896fdd90d362ce27cbeee0a5492180c5e2a6d77810366" exitCode=0
Sep 15 16:44:53.800462 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:53.800136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" event={"ID":"dc8d2de2-37f2-4e49-93a2-0580570e87c2","Type":"ContainerDied","Data":"43f6f4c43e2b7f876bc896fdd90d362ce27cbeee0a5492180c5e2a6d77810366"}
Sep 15 16:44:53.818505 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:53.818485    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:53.908448 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:53.908387    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8d2de2-37f2-4e49-93a2-0580570e87c2-openshift-service-ca-bundle\") pod \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") "
Sep 15 16:44:53.908647 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:53.908474    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls\") pod \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\" (UID: \"dc8d2de2-37f2-4e49-93a2-0580570e87c2\") "
Sep 15 16:44:53.908717 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:53.908692    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8d2de2-37f2-4e49-93a2-0580570e87c2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "dc8d2de2-37f2-4e49-93a2-0580570e87c2" (UID: "dc8d2de2-37f2-4e49-93a2-0580570e87c2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:53.910556 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:53.910526    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dc8d2de2-37f2-4e49-93a2-0580570e87c2" (UID: "dc8d2de2-37f2-4e49-93a2-0580570e87c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:44:54.009590 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:54.009560    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dc8d2de2-37f2-4e49-93a2-0580570e87c2-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:54.009738 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:54.009594    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc8d2de2-37f2-4e49-93a2-0580570e87c2-openshift-service-ca-bundle\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:44:54.806168 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:54.806136    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"
Sep 15 16:44:54.806637 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:54.806168    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf" event={"ID":"dc8d2de2-37f2-4e49-93a2-0580570e87c2","Type":"ContainerDied","Data":"23954f7198c2a853391bcb4666b1238f459447c4dee01490982fac8fca35a9b4"}
Sep 15 16:44:54.806637 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:54.806225    2564 scope.go:117] "RemoveContainer" containerID="43f6f4c43e2b7f876bc896fdd90d362ce27cbeee0a5492180c5e2a6d77810366"
Sep 15 16:44:54.821213 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:54.821192    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"]
Sep 15 16:44:54.825156 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:54.825138    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-da39c-5b77f48bbd-g9mqf"]
Sep 15 16:44:56.163811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:56.163778    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" path="/var/lib/kubelet/pods/dc8d2de2-37f2-4e49-93a2-0580570e87c2/volumes"
Sep 15 16:44:56.516456 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:56.516423    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:44:57.760228 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:57.760184    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:44:57.760756 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:44:57.760733    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:45:01.159302 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:01.159264    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:45:07.760283 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:07.760236    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:45:07.760945 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:07.760653    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:45:11.160205 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:11.160175    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:45:17.760452 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:17.760382    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:45:17.760968 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:17.760863    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:45:27.760723 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:27.760679    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:45:27.761255 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:27.761141    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:45:28.882674 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.882640    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"]
Sep 15 16:45:28.883029 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.882953    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c"
Sep 15 16:45:28.883029 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.882965    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c"
Sep 15 16:45:28.883098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.883035    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc8d2de2-37f2-4e49-93a2-0580570e87c2" containerName="model-chainer-raw-hpa-da39c"
Sep 15 16:45:28.886461 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.886438    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:28.888708 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.888688    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-97930a-dockercfg-ctxc4\""
Sep 15 16:45:28.888806 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.888690    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-97930a-kube-rbac-proxy-sar-config\""
Sep 15 16:45:28.888859 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.888845    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-97930a\""
Sep 15 16:45:28.888908 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.888876    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-97930a-predictor-serving-cert\""
Sep 15 16:45:28.896031 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.896012    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"]
Sep 15 16:45:28.993703 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.993675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:28.993896 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.993743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-cabundle-cert\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:28.993896 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.993781    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7553f0e5-4098-4f0a-987f-f00406d5b615-kserve-provision-location\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:28.993896 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.993818    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmrll\" (UniqueName: \"kubernetes.io/projected/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-api-access-rmrll\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:28.993896 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:28.993860    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.095202 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.095165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-cabundle-cert\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.095446 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.095217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7553f0e5-4098-4f0a-987f-f00406d5b615-kserve-provision-location\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.095446 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.095244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmrll\" (UniqueName: \"kubernetes.io/projected/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-api-access-rmrll\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.095446 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.095262    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.095446 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.095303    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.095446 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:29.095433    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-97930a-predictor-serving-cert: secret "isvc-secondary-97930a-predictor-serving-cert" not found
Sep 15 16:45:29.095804 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:29.095514    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls podName:7553f0e5-4098-4f0a-987f-f00406d5b615 nodeName:}" failed. No retries permitted until 2026-09-15 16:45:29.595482706 +0000 UTC m=+1133.931437635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls") pod "isvc-secondary-97930a-predictor-66c799cd9f-qx2fv" (UID: "7553f0e5-4098-4f0a-987f-f00406d5b615") : secret "isvc-secondary-97930a-predictor-serving-cert" not found
Sep 15 16:45:29.095804 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.095706    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7553f0e5-4098-4f0a-987f-f00406d5b615-kserve-provision-location\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.096004 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.095982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-cabundle-cert\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.096084 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.096005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.103919 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.103898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmrll\" (UniqueName: \"kubernetes.io/projected/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-api-access-rmrll\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.599723 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.599688    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.602186 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.602156    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls\") pod \"isvc-secondary-97930a-predictor-66c799cd9f-qx2fv\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") " pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:29.797353 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:29.797319    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:30.067307 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:30.067271    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"]
Sep 15 16:45:30.070318 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:45:30.070296    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7553f0e5_4098_4f0a_987f_f00406d5b615.slice/crio-8227c7d213beba4243b2f7ce17b72b1aa2ab171e2cb6710d4338986994b18cbb WatchSource:0}: Error finding container 8227c7d213beba4243b2f7ce17b72b1aa2ab171e2cb6710d4338986994b18cbb: Status 404 returned error can't find the container with id 8227c7d213beba4243b2f7ce17b72b1aa2ab171e2cb6710d4338986994b18cbb
Sep 15 16:45:30.948070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:30.948032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv" event={"ID":"7553f0e5-4098-4f0a-987f-f00406d5b615","Type":"ContainerStarted","Data":"855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6"}
Sep 15 16:45:30.948274 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:30.948116    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv" event={"ID":"7553f0e5-4098-4f0a-987f-f00406d5b615","Type":"ContainerStarted","Data":"8227c7d213beba4243b2f7ce17b72b1aa2ab171e2cb6710d4338986994b18cbb"}
Sep 15 16:45:35.969320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:35.969252    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_7553f0e5-4098-4f0a-987f-f00406d5b615/storage-initializer/0.log"
Sep 15 16:45:35.969320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:35.969286    2564 generic.go:356] "Generic (PLEG): container finished" podID="7553f0e5-4098-4f0a-987f-f00406d5b615" containerID="855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6" exitCode=1
Sep 15 16:45:35.969758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:35.969366    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv" event={"ID":"7553f0e5-4098-4f0a-987f-f00406d5b615","Type":"ContainerDied","Data":"855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6"}
Sep 15 16:45:36.974936 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:36.974905    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_7553f0e5-4098-4f0a-987f-f00406d5b615/storage-initializer/0.log"
Sep 15 16:45:36.975305 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:36.975031    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv" event={"ID":"7553f0e5-4098-4f0a-987f-f00406d5b615","Type":"ContainerStarted","Data":"c2e503d4e7ff03bc1495e82f27fb2d0897231ea2bbcf23951d6646ea08c94b54"}
Sep 15 16:45:37.760281 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:37.760231    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:45:37.760709 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:37.760677    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:45:41.996011 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:41.995986    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_7553f0e5-4098-4f0a-987f-f00406d5b615/storage-initializer/1.log"
Sep 15 16:45:41.996491 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:41.996476    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_7553f0e5-4098-4f0a-987f-f00406d5b615/storage-initializer/0.log"
Sep 15 16:45:41.996541 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:41.996508    2564 generic.go:356] "Generic (PLEG): container finished" podID="7553f0e5-4098-4f0a-987f-f00406d5b615" containerID="c2e503d4e7ff03bc1495e82f27fb2d0897231ea2bbcf23951d6646ea08c94b54" exitCode=1
Sep 15 16:45:41.996590 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:41.996579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv" event={"ID":"7553f0e5-4098-4f0a-987f-f00406d5b615","Type":"ContainerDied","Data":"c2e503d4e7ff03bc1495e82f27fb2d0897231ea2bbcf23951d6646ea08c94b54"}
Sep 15 16:45:41.996662 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:41.996653    2564 scope.go:117] "RemoveContainer" containerID="855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6"
Sep 15 16:45:41.997014 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:41.996996    2564 scope.go:117] "RemoveContainer" containerID="855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6"
Sep 15 16:45:42.006989 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:42.006970    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_kserve-ci-e2e-test(7553f0e5-4098-4f0a-987f-f00406d5b615)\"" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv" podUID="7553f0e5-4098-4f0a-987f-f00406d5b615"
Sep 15 16:45:42.007244 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:42.007220    2564 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_kserve-ci-e2e-test_7553f0e5-4098-4f0a-987f-f00406d5b615_0 in pod sandbox 8227c7d213beba4243b2f7ce17b72b1aa2ab171e2cb6710d4338986994b18cbb from index: no such id: '855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6'" containerID="855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6"
Sep 15 16:45:42.007296 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:42.007253    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_kserve-ci-e2e-test_7553f0e5-4098-4f0a-987f-f00406d5b615_0 in pod sandbox 8227c7d213beba4243b2f7ce17b72b1aa2ab171e2cb6710d4338986994b18cbb from index: no such id: '855c66bdf4d806551287c1730e8de5c07c81a829c9c27f00cf193a9abb3018c6'"
Sep 15 16:45:43.001816 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:43.001790    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_7553f0e5-4098-4f0a-987f-f00406d5b615/storage-initializer/1.log"
Sep 15 16:45:46.901276 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:46.901246    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"]
Sep 15 16:45:46.933650 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:46.933620    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"]
Sep 15 16:45:46.934173 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:46.934140    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" containerID="cri-o://6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5" gracePeriod=30
Sep 15 16:45:46.934388 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:46.934216    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kube-rbac-proxy" containerID="cri-o://46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046" gracePeriod=30
Sep 15 16:45:47.054507 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.054489    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_7553f0e5-4098-4f0a-987f-f00406d5b615/storage-initializer/1.log"
Sep 15 16:45:47.056622 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.056607    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:47.248064 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.247972    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7553f0e5-4098-4f0a-987f-f00406d5b615-kserve-provision-location\") pod \"7553f0e5-4098-4f0a-987f-f00406d5b615\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") "
Sep 15 16:45:47.248064 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.248040    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-rbac-proxy-sar-config\") pod \"7553f0e5-4098-4f0a-987f-f00406d5b615\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") "
Sep 15 16:45:47.248293 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.248069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmrll\" (UniqueName: \"kubernetes.io/projected/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-api-access-rmrll\") pod \"7553f0e5-4098-4f0a-987f-f00406d5b615\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") "
Sep 15 16:45:47.248293 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.248096    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-cabundle-cert\") pod \"7553f0e5-4098-4f0a-987f-f00406d5b615\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") "
Sep 15 16:45:47.248293 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.248166    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls\") pod \"7553f0e5-4098-4f0a-987f-f00406d5b615\" (UID: \"7553f0e5-4098-4f0a-987f-f00406d5b615\") "
Sep 15 16:45:47.248488 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.248306    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7553f0e5-4098-4f0a-987f-f00406d5b615-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7553f0e5-4098-4f0a-987f-f00406d5b615" (UID: "7553f0e5-4098-4f0a-987f-f00406d5b615"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:45:47.248548 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.248527    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7553f0e5-4098-4f0a-987f-f00406d5b615" (UID: "7553f0e5-4098-4f0a-987f-f00406d5b615"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:45:47.248824 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.248800    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7553f0e5-4098-4f0a-987f-f00406d5b615" (UID: "7553f0e5-4098-4f0a-987f-f00406d5b615"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:45:47.250497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.250469    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7553f0e5-4098-4f0a-987f-f00406d5b615" (UID: "7553f0e5-4098-4f0a-987f-f00406d5b615"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:45:47.250858 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.250832    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-api-access-rmrll" (OuterVolumeSpecName: "kube-api-access-rmrll") pod "7553f0e5-4098-4f0a-987f-f00406d5b615" (UID: "7553f0e5-4098-4f0a-987f-f00406d5b615"). InnerVolumeSpecName "kube-api-access-rmrll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:45:47.348993 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.348950    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmrll\" (UniqueName: \"kubernetes.io/projected/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-api-access-rmrll\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:47.348993 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.348979    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:47.348993 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.348989    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7553f0e5-4098-4f0a-987f-f00406d5b615-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:47.348993 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.348998    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7553f0e5-4098-4f0a-987f-f00406d5b615-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:47.349323 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.349007    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7553f0e5-4098-4f0a-987f-f00406d5b615-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:47.760109 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.760069    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:45:47.760606 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:47.760580    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:45:48.022639 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.022559    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-97930a-predictor-66c799cd9f-qx2fv_7553f0e5-4098-4f0a-987f-f00406d5b615/storage-initializer/1.log"
Sep 15 16:45:48.023067 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.022657    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"
Sep 15 16:45:48.023067 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.022671    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv" event={"ID":"7553f0e5-4098-4f0a-987f-f00406d5b615","Type":"ContainerDied","Data":"8227c7d213beba4243b2f7ce17b72b1aa2ab171e2cb6710d4338986994b18cbb"}
Sep 15 16:45:48.023067 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.022725    2564 scope.go:117] "RemoveContainer" containerID="c2e503d4e7ff03bc1495e82f27fb2d0897231ea2bbcf23951d6646ea08c94b54"
Sep 15 16:45:48.025035 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.025009    2564 generic.go:356] "Generic (PLEG): container finished" podID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerID="46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046" exitCode=2
Sep 15 16:45:48.025184 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.025058    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" event={"ID":"432d4def-10d4-4ca9-9b8a-d6b48d6134f9","Type":"ContainerDied","Data":"46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046"}
Sep 15 16:45:48.059952 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.059921    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"]
Sep 15 16:45:48.063923 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.063898    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-97930a-predictor-66c799cd9f-qx2fv"]
Sep 15 16:45:48.164796 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:48.164759    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7553f0e5-4098-4f0a-987f-f00406d5b615" path="/var/lib/kubelet/pods/7553f0e5-4098-4f0a-987f-f00406d5b615/volumes"
Sep 15 16:45:51.159893 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.159856    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 15 16:45:51.511899 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.511859    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 15 16:45:51.684826 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.684800    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:45:51.685090 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685069    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-rbac-proxy-sar-config\") pod \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") "
Sep 15 16:45:51.685168 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685154    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls\") pod \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") "
Sep 15 16:45:51.685229 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685215    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kserve-provision-location\") pod \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") "
Sep 15 16:45:51.685351 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685318    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-cabundle-cert\") pod \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") "
Sep 15 16:45:51.685508 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685378    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wmx4\" (UniqueName: \"kubernetes.io/projected/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-api-access-4wmx4\") pod \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\" (UID: \"432d4def-10d4-4ca9-9b8a-d6b48d6134f9\") "
Sep 15 16:45:51.685508 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685432    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "432d4def-10d4-4ca9-9b8a-d6b48d6134f9" (UID: "432d4def-10d4-4ca9-9b8a-d6b48d6134f9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:45:51.685508 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685478    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "432d4def-10d4-4ca9-9b8a-d6b48d6134f9" (UID: "432d4def-10d4-4ca9-9b8a-d6b48d6134f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:45:51.685663 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685642    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:51.685707 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685663    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:51.685707 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.685668    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "432d4def-10d4-4ca9-9b8a-d6b48d6134f9" (UID: "432d4def-10d4-4ca9-9b8a-d6b48d6134f9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:45:51.687315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.687298    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "432d4def-10d4-4ca9-9b8a-d6b48d6134f9" (UID: "432d4def-10d4-4ca9-9b8a-d6b48d6134f9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:45:51.687445 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.687418    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-api-access-4wmx4" (OuterVolumeSpecName: "kube-api-access-4wmx4") pod "432d4def-10d4-4ca9-9b8a-d6b48d6134f9" (UID: "432d4def-10d4-4ca9-9b8a-d6b48d6134f9"). InnerVolumeSpecName "kube-api-access-4wmx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:45:51.786359 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.786328    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:51.786359 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.786356    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wmx4\" (UniqueName: \"kubernetes.io/projected/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-kube-api-access-4wmx4\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:51.786588 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:51.786367    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/432d4def-10d4-4ca9-9b8a-d6b48d6134f9-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:45:52.044282 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.044191    2564 generic.go:356] "Generic (PLEG): container finished" podID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerID="6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5" exitCode=0
Sep 15 16:45:52.044282 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.044259    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"
Sep 15 16:45:52.044525 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.044261    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" event={"ID":"432d4def-10d4-4ca9-9b8a-d6b48d6134f9","Type":"ContainerDied","Data":"6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5"}
Sep 15 16:45:52.044525 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.044461    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c" event={"ID":"432d4def-10d4-4ca9-9b8a-d6b48d6134f9","Type":"ContainerDied","Data":"530da0371fa4b75c946624d438f523d05a50dfa03db644808fa95b06b51bca51"}
Sep 15 16:45:52.044525 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.044515    2564 scope.go:117] "RemoveContainer" containerID="46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046"
Sep 15 16:45:52.053713 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.053693    2564 scope.go:117] "RemoveContainer" containerID="6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5"
Sep 15 16:45:52.062779 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.062760    2564 scope.go:117] "RemoveContainer" containerID="2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210"
Sep 15 16:45:52.070045 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.070020    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"]
Sep 15 16:45:52.071406 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.071372    2564 scope.go:117] "RemoveContainer" containerID="46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046"
Sep 15 16:45:52.071743 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:52.071715    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046\": container with ID starting with 46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046 not found: ID does not exist" containerID="46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046"
Sep 15 16:45:52.071796 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.071759    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046"} err="failed to get container status \"46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046\": rpc error: code = NotFound desc = could not find container \"46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046\": container with ID starting with 46f1a9f0082f8f24f874421f7ec7f03ff59c03d160646c9f36dc12f7f3013046 not found: ID does not exist"
Sep 15 16:45:52.071796 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.071785    2564 scope.go:117] "RemoveContainer" containerID="6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5"
Sep 15 16:45:52.072119 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:52.072097    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5\": container with ID starting with 6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5 not found: ID does not exist" containerID="6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5"
Sep 15 16:45:52.072183 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.072127    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5"} err="failed to get container status \"6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5\": rpc error: code = NotFound desc = could not find container \"6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5\": container with ID starting with 6ab605c5fb8c14a8564602bb267da3c8132a7e32ee5b92e528a2fa6c7dba58f5 not found: ID does not exist"
Sep 15 16:45:52.072183 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.072143    2564 scope.go:117] "RemoveContainer" containerID="2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210"
Sep 15 16:45:52.072444 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:52.072425    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210\": container with ID starting with 2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210 not found: ID does not exist" containerID="2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210"
Sep 15 16:45:52.072504 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.072451    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210"} err="failed to get container status \"2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210\": rpc error: code = NotFound desc = could not find container \"2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210\": container with ID starting with 2dbadb8ccb341aa8e5ab48467a91daf4a4cea2ccb13001526001087111aec210 not found: ID does not exist"
Sep 15 16:45:52.077315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.077295    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-97930a-predictor-75d84c5688-fb25c"]
Sep 15 16:45:52.134886 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.134856    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"]
Sep 15 16:45:52.135195 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135182    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7553f0e5-4098-4f0a-987f-f00406d5b615" containerName="storage-initializer"
Sep 15 16:45:52.135244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135196    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7553f0e5-4098-4f0a-987f-f00406d5b615" containerName="storage-initializer"
Sep 15 16:45:52.135244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135206    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7553f0e5-4098-4f0a-987f-f00406d5b615" containerName="storage-initializer"
Sep 15 16:45:52.135244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135212    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7553f0e5-4098-4f0a-987f-f00406d5b615" containerName="storage-initializer"
Sep 15 16:45:52.135244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135223    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="storage-initializer"
Sep 15 16:45:52.135244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135229    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="storage-initializer"
Sep 15 16:45:52.135244 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135243    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kube-rbac-proxy"
Sep 15 16:45:52.135475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135248    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kube-rbac-proxy"
Sep 15 16:45:52.135475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135255    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container"
Sep 15 16:45:52.135475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135261    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container"
Sep 15 16:45:52.135475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135311    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kube-rbac-proxy"
Sep 15 16:45:52.135475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135319    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7553f0e5-4098-4f0a-987f-f00406d5b615" containerName="storage-initializer"
Sep 15 16:45:52.135475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135324    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7553f0e5-4098-4f0a-987f-f00406d5b615" containerName="storage-initializer"
Sep 15 16:45:52.135475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.135330    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" containerName="kserve-container"
Sep 15 16:45:52.139905 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.139890    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.142277 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.142255    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-be68af-predictor-serving-cert\""
Sep 15 16:45:52.142409 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.142329    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-be68af\""
Sep 15 16:45:52.142409 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.142327    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-be68af-kube-rbac-proxy-sar-config\""
Sep 15 16:45:52.142409 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.142359    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-be68af-dockercfg-9g4j5\""
Sep 15 16:45:52.147487 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.147467    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"]
Sep 15 16:45:52.164704 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.164671    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432d4def-10d4-4ca9-9b8a-d6b48d6134f9" path="/var/lib/kubelet/pods/432d4def-10d4-4ca9-9b8a-d6b48d6134f9/volumes"
Sep 15 16:45:52.290692 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.290651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kserve-provision-location\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.290874 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.290750    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-cabundle-cert\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.290874 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.290810    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.290874 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.290834    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.290874 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.290858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9k4m\" (UniqueName: \"kubernetes.io/projected/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-api-access-k9k4m\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.391795 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.391707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.391795 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.391746    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9k4m\" (UniqueName: \"kubernetes.io/projected/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-api-access-k9k4m\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.391972 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:52.391862    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-serving-cert: secret "isvc-init-fail-be68af-predictor-serving-cert" not found
Sep 15 16:45:52.391972 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:45:52.391920    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls podName:dfce0554-3bfc-4d0a-821c-9b4c7db65ec3 nodeName:}" failed. No retries permitted until 2026-09-15 16:45:52.891903904 +0000 UTC m=+1157.227858813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls") pod "isvc-init-fail-be68af-predictor-69c988c58f-dhmc4" (UID: "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3") : secret "isvc-init-fail-be68af-predictor-serving-cert" not found
Sep 15 16:45:52.391972 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.391917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kserve-provision-location\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.392119 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.392009    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-cabundle-cert\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.392119 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.392047    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.392295 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.392273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kserve-provision-location\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.392690 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.392670    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-cabundle-cert\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.392778 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.392693    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.401302 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.401280    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9k4m\" (UniqueName: \"kubernetes.io/projected/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-api-access-k9k4m\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.896172 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.896118    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:52.898724 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:52.898704    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls\") pod \"isvc-init-fail-be68af-predictor-69c988c58f-dhmc4\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") " pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:53.050784 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:53.050754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:45:53.321976 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:53.321943    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"]
Sep 15 16:45:53.324291 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:45:53.324264    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfce0554_3bfc_4d0a_821c_9b4c7db65ec3.slice/crio-9c28cb023ecf4102813e10605753a4bc2da63aacaee611d50a20ce1d7febaaf7 WatchSource:0}: Error finding container 9c28cb023ecf4102813e10605753a4bc2da63aacaee611d50a20ce1d7febaaf7: Status 404 returned error can't find the container with id 9c28cb023ecf4102813e10605753a4bc2da63aacaee611d50a20ce1d7febaaf7
Sep 15 16:45:54.056509 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:54.056476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4" event={"ID":"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3","Type":"ContainerStarted","Data":"a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4"}
Sep 15 16:45:54.056698 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:54.056541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4" event={"ID":"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3","Type":"ContainerStarted","Data":"9c28cb023ecf4102813e10605753a4bc2da63aacaee611d50a20ce1d7febaaf7"}
Sep 15 16:45:57.071851 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:57.071823    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-be68af-predictor-69c988c58f-dhmc4_dfce0554-3bfc-4d0a-821c-9b4c7db65ec3/storage-initializer/0.log"
Sep 15 16:45:57.072290 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:57.071860    2564 generic.go:356] "Generic (PLEG): container finished" podID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerID="a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4" exitCode=1
Sep 15 16:45:57.072290 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:57.071925    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4" event={"ID":"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3","Type":"ContainerDied","Data":"a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4"}
Sep 15 16:45:57.761139 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:57.761110    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:45:57.761336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:57.761223    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:45:58.077666 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:58.077590    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-be68af-predictor-69c988c58f-dhmc4_dfce0554-3bfc-4d0a-821c-9b4c7db65ec3/storage-initializer/0.log"
Sep 15 16:45:58.077666 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:45:58.077637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4" event={"ID":"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3","Type":"ContainerStarted","Data":"6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b"}
Sep 15 16:46:02.106425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.106360    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"]
Sep 15 16:46:02.106907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.106611    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4" podUID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerName="storage-initializer" containerID="cri-o://6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b" gracePeriod=30
Sep 15 16:46:02.552110 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.552086    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-be68af-predictor-69c988c58f-dhmc4_dfce0554-3bfc-4d0a-821c-9b4c7db65ec3/storage-initializer/1.log"
Sep 15 16:46:02.552497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.552479    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-be68af-predictor-69c988c58f-dhmc4_dfce0554-3bfc-4d0a-821c-9b4c7db65ec3/storage-initializer/0.log"
Sep 15 16:46:02.554766 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.554752    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:46:02.566960 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.566934    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kserve-provision-location\") pod \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") "
Sep 15 16:46:02.567067 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.566994    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-rbac-proxy-sar-config\") pod \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") "
Sep 15 16:46:02.567067 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.567035    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9k4m\" (UniqueName: \"kubernetes.io/projected/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-api-access-k9k4m\") pod \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") "
Sep 15 16:46:02.567168 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.567074    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls\") pod \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") "
Sep 15 16:46:02.567168 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.567096    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-cabundle-cert\") pod \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\" (UID: \"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3\") "
Sep 15 16:46:02.567270 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.567220    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" (UID: "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:46:02.567414 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.567356    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" (UID: "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:46:02.567579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.567556    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" (UID: "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:46:02.569189 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.569169    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-api-access-k9k4m" (OuterVolumeSpecName: "kube-api-access-k9k4m") pod "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" (UID: "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3"). InnerVolumeSpecName "kube-api-access-k9k4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:46:02.569275 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.569256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" (UID: "dfce0554-3bfc-4d0a-821c-9b4c7db65ec3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:46:02.668584 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.668487    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9k4m\" (UniqueName: \"kubernetes.io/projected/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-api-access-k9k4m\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:02.668584 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.668520    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:02.668584 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.668529    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:02.668584 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.668537    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:02.668584 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:02.668545    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:03.105330 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.105304    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-be68af-predictor-69c988c58f-dhmc4_dfce0554-3bfc-4d0a-821c-9b4c7db65ec3/storage-initializer/1.log"
Sep 15 16:46:03.105750 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.105734    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-be68af-predictor-69c988c58f-dhmc4_dfce0554-3bfc-4d0a-821c-9b4c7db65ec3/storage-initializer/0.log"
Sep 15 16:46:03.105811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.105770    2564 generic.go:356] "Generic (PLEG): container finished" podID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerID="6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b" exitCode=1
Sep 15 16:46:03.105862 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.105824    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"
Sep 15 16:46:03.105862 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.105835    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4" event={"ID":"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3","Type":"ContainerDied","Data":"6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b"}
Sep 15 16:46:03.105938 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.105884    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4" event={"ID":"dfce0554-3bfc-4d0a-821c-9b4c7db65ec3","Type":"ContainerDied","Data":"9c28cb023ecf4102813e10605753a4bc2da63aacaee611d50a20ce1d7febaaf7"}
Sep 15 16:46:03.105938 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.105910    2564 scope.go:117] "RemoveContainer" containerID="6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b"
Sep 15 16:46:03.115120 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.114926    2564 scope.go:117] "RemoveContainer" containerID="a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4"
Sep 15 16:46:03.122950 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.122926    2564 scope.go:117] "RemoveContainer" containerID="6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b"
Sep 15 16:46:03.123217 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:46:03.123197    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b\": container with ID starting with 6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b not found: ID does not exist" containerID="6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b"
Sep 15 16:46:03.123273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.123227    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b"} err="failed to get container status \"6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b\": rpc error: code = NotFound desc = could not find container \"6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b\": container with ID starting with 6e04285ec890466b42c89aa31e8516cc472e278ffc4c91e9585e69fae7ef998b not found: ID does not exist"
Sep 15 16:46:03.123273 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.123244    2564 scope.go:117] "RemoveContainer" containerID="a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4"
Sep 15 16:46:03.123537 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:46:03.123519    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4\": container with ID starting with a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4 not found: ID does not exist" containerID="a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4"
Sep 15 16:46:03.123602 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.123547    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4"} err="failed to get container status \"a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4\": rpc error: code = NotFound desc = could not find container \"a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4\": container with ID starting with a1d99277c64fe768de743e9549e14fd781b2ecc561ec6fa901439ec004cb76f4 not found: ID does not exist"
Sep 15 16:46:03.140360 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.140327    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"]
Sep 15 16:46:03.143889 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:03.143864    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-be68af-predictor-69c988c58f-dhmc4"]
Sep 15 16:46:04.164239 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:04.164206    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" path="/var/lib/kubelet/pods/dfce0554-3bfc-4d0a-821c-9b4c7db65ec3/volumes"
Sep 15 16:46:07.346906 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.346873    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"]
Sep 15 16:46:07.347272 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.347229    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerName="storage-initializer"
Sep 15 16:46:07.347272 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.347240    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerName="storage-initializer"
Sep 15 16:46:07.347272 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.347256    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerName="storage-initializer"
Sep 15 16:46:07.347272 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.347262    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerName="storage-initializer"
Sep 15 16:46:07.347439 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.347318    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerName="storage-initializer"
Sep 15 16:46:07.347439 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.347327    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfce0554-3bfc-4d0a-821c-9b4c7db65ec3" containerName="storage-initializer"
Sep 15 16:46:07.351662 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.351647    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.354255 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.354235    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-efad6-predictor-serving-cert\""
Sep 15 16:46:07.354379 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.354289    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-efad6-kube-rbac-proxy-sar-config\""
Sep 15 16:46:07.359867 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.359832    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"]
Sep 15 16:46:07.404410 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.404370    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39aabda6-7324-465c-995c-22778b6cdef3-proxy-tls\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.404561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.404443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.404561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.404465    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39aabda6-7324-465c-995c-22778b6cdef3-kserve-provision-location\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.404561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.404485    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x48cq\" (UniqueName: \"kubernetes.io/projected/39aabda6-7324-465c-995c-22778b6cdef3-kube-api-access-x48cq\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.404561 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.404510    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-cabundle-cert\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.505965 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.505929    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39aabda6-7324-465c-995c-22778b6cdef3-proxy-tls\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.505965 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.505969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.506190 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.505988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39aabda6-7324-465c-995c-22778b6cdef3-kserve-provision-location\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.506249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.506167    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x48cq\" (UniqueName: \"kubernetes.io/projected/39aabda6-7324-465c-995c-22778b6cdef3-kube-api-access-x48cq\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.506249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.506222    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-cabundle-cert\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.506490 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.506467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39aabda6-7324-465c-995c-22778b6cdef3-kserve-provision-location\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.506760 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.506745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.506866 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.506849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-cabundle-cert\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.508634 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.508614    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39aabda6-7324-465c-995c-22778b6cdef3-proxy-tls\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.514545 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.514479    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x48cq\" (UniqueName: \"kubernetes.io/projected/39aabda6-7324-465c-995c-22778b6cdef3-kube-api-access-x48cq\") pod \"raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") " pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.662221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.662136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:07.931904 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:07.931879    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"]
Sep 15 16:46:07.934202 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:46:07.934179    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39aabda6_7324_465c_995c_22778b6cdef3.slice/crio-233213edae85f44ec1c6e9037c1194a5bef32a3384ab3e6469b2aa883822fcbe WatchSource:0}: Error finding container 233213edae85f44ec1c6e9037c1194a5bef32a3384ab3e6469b2aa883822fcbe: Status 404 returned error can't find the container with id 233213edae85f44ec1c6e9037c1194a5bef32a3384ab3e6469b2aa883822fcbe
Sep 15 16:46:08.130362 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:08.130330    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" event={"ID":"39aabda6-7324-465c-995c-22778b6cdef3","Type":"ContainerStarted","Data":"9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe"}
Sep 15 16:46:08.130544 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:08.130411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" event={"ID":"39aabda6-7324-465c-995c-22778b6cdef3","Type":"ContainerStarted","Data":"233213edae85f44ec1c6e9037c1194a5bef32a3384ab3e6469b2aa883822fcbe"}
Sep 15 16:46:09.087192 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:09.087158    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"]
Sep 15 16:46:09.087591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:09.087503    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" containerID="cri-o://2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e" gracePeriod=30
Sep 15 16:46:09.087591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:09.087523    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" containerID="cri-o://699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8" gracePeriod=30
Sep 15 16:46:09.087591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:09.087565    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy" containerID="cri-o://23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950" gracePeriod=30
Sep 15 16:46:10.140491 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:10.140459    2564 generic.go:356] "Generic (PLEG): container finished" podID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerID="23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950" exitCode=2
Sep 15 16:46:10.140862 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:10.140520    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerDied","Data":"23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950"}
Sep 15 16:46:12.151866 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:12.151779    2564 generic.go:356] "Generic (PLEG): container finished" podID="39aabda6-7324-465c-995c-22778b6cdef3" containerID="9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe" exitCode=0
Sep 15 16:46:12.152236 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:12.151860    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" event={"ID":"39aabda6-7324-465c-995c-22778b6cdef3","Type":"ContainerDied","Data":"9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe"}
Sep 15 16:46:12.755643 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:12.755601    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 15 16:46:13.158492 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:13.158414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" event={"ID":"39aabda6-7324-465c-995c-22778b6cdef3","Type":"ContainerStarted","Data":"492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f"}
Sep 15 16:46:13.158492 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:13.158476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" event={"ID":"39aabda6-7324-465c-995c-22778b6cdef3","Type":"ContainerStarted","Data":"0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126"}
Sep 15 16:46:13.158907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:13.158674    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:13.178314 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:13.178266    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podStartSLOduration=2.342587595 podStartE2EDuration="6.178249481s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835661886s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:46:13.176523839 +0000 UTC m=+1177.512478771" watchObservedRunningTime="2026-09-15 16:46:13.178249481 +0000 UTC m=+1177.514204451"
Sep 15 16:46:14.166652 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:14.166621    2564 generic.go:356] "Generic (PLEG): container finished" podID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerID="2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e" exitCode=0
Sep 15 16:46:14.167099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:14.166685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerDied","Data":"2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e"}
Sep 15 16:46:14.167099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:14.167045    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:14.168213 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:14.168189    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:46:15.170842 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:15.170807    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:46:17.756224 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:17.756177    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 15 16:46:17.760522 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:17.760497    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:46:17.760971 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:17.760951    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:46:20.174905 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:20.174880    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:46:20.175350 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:20.175313    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:46:22.755568 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:22.755524    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 15 16:46:22.755936 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:22.755684    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:46:27.755725 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:27.755678    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 15 16:46:27.760046 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:27.760020    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:46:27.760380 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:27.760353    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:46:30.175828 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:30.175791    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:46:32.755778 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:32.755738    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 15 16:46:37.755460 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:37.755385    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 15 16:46:37.760413 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:37.760366    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 15 16:46:37.760567 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:37.760512    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:46:37.760835 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:37.760809    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:46:37.760945 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:37.760926    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:46:39.229925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.229903    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:46:39.272416 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.272367    2564 generic.go:356] "Generic (PLEG): container finished" podID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerID="699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8" exitCode=0
Sep 15 16:46:39.272579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.272441    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"
Sep 15 16:46:39.272579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.272457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerDied","Data":"699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8"}
Sep 15 16:46:39.272579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.272522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw" event={"ID":"a5d193ac-709a-4fe3-a07a-202712e0d8b9","Type":"ContainerDied","Data":"af38d4783fc2a8234cd6b411dd3ebbd314ff63adf1f2ba89dc48871016002ca3"}
Sep 15 16:46:39.272579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.272552    2564 scope.go:117] "RemoveContainer" containerID="699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8"
Sep 15 16:46:39.280329 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.280313    2564 scope.go:117] "RemoveContainer" containerID="23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950"
Sep 15 16:46:39.287486 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.287466    2564 scope.go:117] "RemoveContainer" containerID="2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e"
Sep 15 16:46:39.294435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.294407    2564 scope.go:117] "RemoveContainer" containerID="896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b"
Sep 15 16:46:39.301107 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.301092    2564 scope.go:117] "RemoveContainer" containerID="699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8"
Sep 15 16:46:39.301405 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:46:39.301370    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8\": container with ID starting with 699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8 not found: ID does not exist" containerID="699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8"
Sep 15 16:46:39.301475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.301413    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8"} err="failed to get container status \"699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8\": rpc error: code = NotFound desc = could not find container \"699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8\": container with ID starting with 699ed885728f37eed69d6a21642bb346550ec027694099ec910440b2ebe2b3f8 not found: ID does not exist"
Sep 15 16:46:39.301475 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.301430    2564 scope.go:117] "RemoveContainer" containerID="23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950"
Sep 15 16:46:39.301706 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:46:39.301691    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950\": container with ID starting with 23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950 not found: ID does not exist" containerID="23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950"
Sep 15 16:46:39.301754 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.301709    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950"} err="failed to get container status \"23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950\": rpc error: code = NotFound desc = could not find container \"23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950\": container with ID starting with 23a712f278728e3b02a8ff9f8c47ec6ee8613b028ebf2b4d57d448e801927950 not found: ID does not exist"
Sep 15 16:46:39.301754 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.301722    2564 scope.go:117] "RemoveContainer" containerID="2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e"
Sep 15 16:46:39.301951 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:46:39.301935    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e\": container with ID starting with 2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e not found: ID does not exist" containerID="2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e"
Sep 15 16:46:39.301992 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.301957    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e"} err="failed to get container status \"2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e\": rpc error: code = NotFound desc = could not find container \"2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e\": container with ID starting with 2785841af18ec2846096ab31e5f4d7c8e17c1df780e07fd0ca17170e2b287b9e not found: ID does not exist"
Sep 15 16:46:39.301992 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.301972    2564 scope.go:117] "RemoveContainer" containerID="896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b"
Sep 15 16:46:39.302210 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:46:39.302195    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b\": container with ID starting with 896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b not found: ID does not exist" containerID="896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b"
Sep 15 16:46:39.302250 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.302215    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b"} err="failed to get container status \"896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b\": rpc error: code = NotFound desc = could not find container \"896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b\": container with ID starting with 896e387e09e8d70e9620dcd9dbf3c45ce86cdd6605dab4db3c260297986be21b not found: ID does not exist"
Sep 15 16:46:39.368965 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.368899    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-cabundle-cert\") pod \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") "
Sep 15 16:46:39.368965 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.368959    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kserve-provision-location\") pod \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") "
Sep 15 16:46:39.369125 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.368978    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls\") pod \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") "
Sep 15 16:46:39.369125 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.369013    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-rbac-proxy-sar-config\") pod \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") "
Sep 15 16:46:39.369125 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.369034    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptf57\" (UniqueName: \"kubernetes.io/projected/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-api-access-ptf57\") pod \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\" (UID: \"a5d193ac-709a-4fe3-a07a-202712e0d8b9\") "
Sep 15 16:46:39.369334 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.369311    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5d193ac-709a-4fe3-a07a-202712e0d8b9" (UID: "a5d193ac-709a-4fe3-a07a-202712e0d8b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:46:39.369390 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.369325    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5d193ac-709a-4fe3-a07a-202712e0d8b9" (UID: "a5d193ac-709a-4fe3-a07a-202712e0d8b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:46:39.369390 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.369340    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5d193ac-709a-4fe3-a07a-202712e0d8b9" (UID: "a5d193ac-709a-4fe3-a07a-202712e0d8b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:46:39.371277 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.371255    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5d193ac-709a-4fe3-a07a-202712e0d8b9" (UID: "a5d193ac-709a-4fe3-a07a-202712e0d8b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:46:39.371386 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.371370    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-api-access-ptf57" (OuterVolumeSpecName: "kube-api-access-ptf57") pod "a5d193ac-709a-4fe3-a07a-202712e0d8b9" (UID: "a5d193ac-709a-4fe3-a07a-202712e0d8b9"). InnerVolumeSpecName "kube-api-access-ptf57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:46:39.470618 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.470577    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:39.470618 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.470613    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:39.470618 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.470624    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5d193ac-709a-4fe3-a07a-202712e0d8b9-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:39.470842 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.470633    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:39.470842 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.470642    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptf57\" (UniqueName: \"kubernetes.io/projected/a5d193ac-709a-4fe3-a07a-202712e0d8b9-kube-api-access-ptf57\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:46:39.593134 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.593105    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"]
Sep 15 16:46:39.597487 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:39.597463    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-ae4ee-predictor-65fdfbb7dd-jcjzw"]
Sep 15 16:46:40.164664 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:40.164636    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" path="/var/lib/kubelet/pods/a5d193ac-709a-4fe3-a07a-202712e0d8b9/volumes"
Sep 15 16:46:40.175320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:40.175290    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:46:50.176151 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:46:50.176109    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:47:00.175768 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:00.175734    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:47:10.175719 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:10.175684    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:47:20.175972 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:20.175933    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:47:30.176572 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:30.176502    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:47:37.423141 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:37.423110    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"]
Sep 15 16:47:37.423578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:37.423334    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" containerID="cri-o://0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126" gracePeriod=30
Sep 15 16:47:37.423578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:37.423379    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kube-rbac-proxy" containerID="cri-o://492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f" gracePeriod=30
Sep 15 16:47:38.498601 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:38.498568    2564 generic.go:356] "Generic (PLEG): container finished" podID="39aabda6-7324-465c-995c-22778b6cdef3" containerID="492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f" exitCode=2
Sep 15 16:47:38.498981 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:38.498613    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" event={"ID":"39aabda6-7324-465c-995c-22778b6cdef3","Type":"ContainerDied","Data":"492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f"}
Sep 15 16:47:40.171995 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:40.171952    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Sep 15 16:47:40.176348 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:40.176316    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 15 16:47:42.163549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.163524    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:47:42.286284 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.286257    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39aabda6-7324-465c-995c-22778b6cdef3-proxy-tls\") pod \"39aabda6-7324-465c-995c-22778b6cdef3\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") "
Sep 15 16:47:42.286457 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.286292    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-kube-rbac-proxy-sar-config\") pod \"39aabda6-7324-465c-995c-22778b6cdef3\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") "
Sep 15 16:47:42.286457 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.286327    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39aabda6-7324-465c-995c-22778b6cdef3-kserve-provision-location\") pod \"39aabda6-7324-465c-995c-22778b6cdef3\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") "
Sep 15 16:47:42.286539 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.286448    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-cabundle-cert\") pod \"39aabda6-7324-465c-995c-22778b6cdef3\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") "
Sep 15 16:47:42.286539 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.286502    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x48cq\" (UniqueName: \"kubernetes.io/projected/39aabda6-7324-465c-995c-22778b6cdef3-kube-api-access-x48cq\") pod \"39aabda6-7324-465c-995c-22778b6cdef3\" (UID: \"39aabda6-7324-465c-995c-22778b6cdef3\") "
Sep 15 16:47:42.286701 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.286678    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39aabda6-7324-465c-995c-22778b6cdef3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39aabda6-7324-465c-995c-22778b6cdef3" (UID: "39aabda6-7324-465c-995c-22778b6cdef3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:47:42.286764 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.286721    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39aabda6-7324-465c-995c-22778b6cdef3" (UID: "39aabda6-7324-465c-995c-22778b6cdef3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:47:42.286845 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.286818    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39aabda6-7324-465c-995c-22778b6cdef3" (UID: "39aabda6-7324-465c-995c-22778b6cdef3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:47:42.288592 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.288559    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39aabda6-7324-465c-995c-22778b6cdef3-kube-api-access-x48cq" (OuterVolumeSpecName: "kube-api-access-x48cq") pod "39aabda6-7324-465c-995c-22778b6cdef3" (UID: "39aabda6-7324-465c-995c-22778b6cdef3"). InnerVolumeSpecName "kube-api-access-x48cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:47:42.288671 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.288599    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39aabda6-7324-465c-995c-22778b6cdef3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39aabda6-7324-465c-995c-22778b6cdef3" (UID: "39aabda6-7324-465c-995c-22778b6cdef3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:47:42.387852 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.387798    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:47:42.387852 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.387844    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x48cq\" (UniqueName: \"kubernetes.io/projected/39aabda6-7324-465c-995c-22778b6cdef3-kube-api-access-x48cq\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:47:42.387852 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.387859    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39aabda6-7324-465c-995c-22778b6cdef3-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:47:42.388095 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.387872    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39aabda6-7324-465c-995c-22778b6cdef3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:47:42.388095 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.387884    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39aabda6-7324-465c-995c-22778b6cdef3-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:47:42.515075 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.515043    2564 generic.go:356] "Generic (PLEG): container finished" podID="39aabda6-7324-465c-995c-22778b6cdef3" containerID="0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126" exitCode=0
Sep 15 16:47:42.515210 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.515103    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"
Sep 15 16:47:42.515210 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.515119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" event={"ID":"39aabda6-7324-465c-995c-22778b6cdef3","Type":"ContainerDied","Data":"0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126"}
Sep 15 16:47:42.515210 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.515181    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb" event={"ID":"39aabda6-7324-465c-995c-22778b6cdef3","Type":"ContainerDied","Data":"233213edae85f44ec1c6e9037c1194a5bef32a3384ab3e6469b2aa883822fcbe"}
Sep 15 16:47:42.515210 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.515206    2564 scope.go:117] "RemoveContainer" containerID="492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f"
Sep 15 16:47:42.523970 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.523951    2564 scope.go:117] "RemoveContainer" containerID="0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126"
Sep 15 16:47:42.531913 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.531894    2564 scope.go:117] "RemoveContainer" containerID="9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe"
Sep 15 16:47:42.537777 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.537706    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"]
Sep 15 16:47:42.541721 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.541701    2564 scope.go:117] "RemoveContainer" containerID="492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f"
Sep 15 16:47:42.542027 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:47:42.542009    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f\": container with ID starting with 492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f not found: ID does not exist" containerID="492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f"
Sep 15 16:47:42.542108 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.542035    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f"} err="failed to get container status \"492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f\": rpc error: code = NotFound desc = could not find container \"492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f\": container with ID starting with 492046b0ed8ace50a4185c162c5d79dfb4f9df73a657b35df00b8c023c1e261f not found: ID does not exist"
Sep 15 16:47:42.542108 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.542050    2564 scope.go:117] "RemoveContainer" containerID="0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126"
Sep 15 16:47:42.542324 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:47:42.542309    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126\": container with ID starting with 0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126 not found: ID does not exist" containerID="0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126"
Sep 15 16:47:42.542385 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.542328    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126"} err="failed to get container status \"0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126\": rpc error: code = NotFound desc = could not find container \"0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126\": container with ID starting with 0175e31162a78bfbb224cbfc3136c771c716018c7b76ed920bc94d66a1324126 not found: ID does not exist"
Sep 15 16:47:42.542385 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.542341    2564 scope.go:117] "RemoveContainer" containerID="9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe"
Sep 15 16:47:42.542776 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:47:42.542673    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe\": container with ID starting with 9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe not found: ID does not exist" containerID="9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe"
Sep 15 16:47:42.542776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.542737    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe"} err="failed to get container status \"9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe\": rpc error: code = NotFound desc = could not find container \"9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe\": container with ID starting with 9d5e854ebf2f45af8f92696e64b7cdb26da7e799456a5337e75170c9b06baabe not found: ID does not exist"
Sep 15 16:47:42.544499 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.544481    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-efad6-predictor-5d7fd686b5-qwnsb"]
Sep 15 16:47:42.634714 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.634681    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"]
Sep 15 16:47:42.634989 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.634977    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="storage-initializer"
Sep 15 16:47:42.635037 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.634992    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="storage-initializer"
Sep 15 16:47:42.635037 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635004    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container"
Sep 15 16:47:42.635037 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635011    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container"
Sep 15 16:47:42.635037 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635021    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kube-rbac-proxy"
Sep 15 16:47:42.635037 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635026    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kube-rbac-proxy"
Sep 15 16:47:42.635037 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635033    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="storage-initializer"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635040    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="storage-initializer"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635050    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635058    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635067    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635072    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635080    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635085    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635139    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kserve-container"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635148    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kserve-container"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635156    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="kube-rbac-proxy"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635162    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d193ac-709a-4fe3-a07a-202712e0d8b9" containerName="agent"
Sep 15 16:47:42.635217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.635167    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="39aabda6-7324-465c-995c-22778b6cdef3" containerName="kube-rbac-proxy"
Sep 15 16:47:42.639796 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.639776    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.642105 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.642086    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:47:42.642210 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.642144    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:47:42.642210 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.642192    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:47:42.642316 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.642231    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:47:42.642515 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.642500    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lqw4s\""
Sep 15 16:47:42.642622 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.642605    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-d5318-kube-rbac-proxy-sar-config\""
Sep 15 16:47:42.642692 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.642648    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-d5318-predictor-serving-cert\""
Sep 15 16:47:42.654189 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.651839    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"]
Sep 15 16:47:42.791553 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.791454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.791553 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.791503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.791553 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.791525    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kserve-provision-location\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.791823 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.791612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpb84\" (UniqueName: \"kubernetes.io/projected/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-api-access-tpb84\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.791823 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.791725    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-cabundle-cert\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.892673 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.892634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-cabundle-cert\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.892673 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.892681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.892936 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.892717    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.892936 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.892737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kserve-provision-location\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.892936 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.892759    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpb84\" (UniqueName: \"kubernetes.io/projected/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-api-access-tpb84\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.893215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.893191    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kserve-provision-location\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.893462 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.893441    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.893535 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.893456    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-cabundle-cert\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.895253 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.895233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.900937 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.900920    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpb84\" (UniqueName: \"kubernetes.io/projected/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-api-access-tpb84\") pod \"raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:42.955783 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:42.955755    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:43.228961 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:43.228923    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"]
Sep 15 16:47:43.231124 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:47:43.231088    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1497ac79_5ebf_4b1e_84f6_1e71c1d0aaf3.slice/crio-bd25d3ae7180066b40b6052767158b3d13251bc6fa2bc1653f5855ad869a9d21 WatchSource:0}: Error finding container bd25d3ae7180066b40b6052767158b3d13251bc6fa2bc1653f5855ad869a9d21: Status 404 returned error can't find the container with id bd25d3ae7180066b40b6052767158b3d13251bc6fa2bc1653f5855ad869a9d21
Sep 15 16:47:43.521470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:43.521436    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" event={"ID":"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3","Type":"ContainerStarted","Data":"73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499"}
Sep 15 16:47:43.521648 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:43.521499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" event={"ID":"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3","Type":"ContainerStarted","Data":"bd25d3ae7180066b40b6052767158b3d13251bc6fa2bc1653f5855ad869a9d21"}
Sep 15 16:47:44.164322 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:44.164290    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39aabda6-7324-465c-995c-22778b6cdef3" path="/var/lib/kubelet/pods/39aabda6-7324-465c-995c-22778b6cdef3/volumes"
Sep 15 16:47:47.538326 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:47.538297    2564 generic.go:356] "Generic (PLEG): container finished" podID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerID="73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499" exitCode=0
Sep 15 16:47:47.538809 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:47.538375    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" event={"ID":"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3","Type":"ContainerDied","Data":"73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499"}
Sep 15 16:47:48.544436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:48.544378    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" event={"ID":"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3","Type":"ContainerStarted","Data":"40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465"}
Sep 15 16:47:48.544924 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:48.544460    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" event={"ID":"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3","Type":"ContainerStarted","Data":"cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09"}
Sep 15 16:47:48.544924 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:48.544626    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:48.566991 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:48.566939    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podStartSLOduration=2.681969435 podStartE2EDuration="6.566923753s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.884954318s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:47:48.564122565 +0000 UTC m=+1272.900077509" watchObservedRunningTime="2026-09-15 16:47:48.566923753 +0000 UTC m=+1272.902878685"
Sep 15 16:47:49.549270 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:49.549240    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:49.550685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:49.550655    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:47:50.553692 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:50.553652    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:47:55.558739 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:55.558710    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:47:55.559385 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:47:55.559357    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:05.559780 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:05.559743    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:15.560139 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:15.560100    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:25.559311 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:25.559271    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:35.560080 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:35.560037    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:44.460604 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.460568    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"]
Sep 15 16:48:44.465441 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.465417    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.467780 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.467758    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 15 16:48:44.468025 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.468009    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 15 16:48:44.473950 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.473928    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"]
Sep 15 16:48:44.502288 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.502255    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"]
Sep 15 16:48:44.505594 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.505578    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.507861 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.507841    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 15 16:48:44.515930 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.515907    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"]
Sep 15 16:48:44.601568 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601529    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa360b6-12a9-4418-b3f2-722c43843310-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.601568 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.601776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.601776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.601776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601742    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xsd\" (UniqueName: \"kubernetes.io/projected/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-api-access-z2xsd\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.601776 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.601910 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn7nv\" (UniqueName: \"kubernetes.io/projected/7aa360b6-12a9-4418-b3f2-722c43843310-kube-api-access-kn7nv\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.601910 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601804    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17345f0b-eb61-4bdf-9659-b0c78fb3967c-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.601910 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601879    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.601910 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.601899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa360b6-12a9-4418-b3f2-722c43843310-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.702425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702375    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xsd\" (UniqueName: \"kubernetes.io/projected/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-api-access-z2xsd\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.702425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702421    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.702685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702443    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn7nv\" (UniqueName: \"kubernetes.io/projected/7aa360b6-12a9-4418-b3f2-722c43843310-kube-api-access-kn7nv\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.702685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17345f0b-eb61-4bdf-9659-b0c78fb3967c-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.702685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.702685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa360b6-12a9-4418-b3f2-722c43843310-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.702685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702557    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa360b6-12a9-4418-b3f2-722c43843310-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.702685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702577    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.702685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702608    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.702685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.702641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.703120 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.703010    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa360b6-12a9-4418-b3f2-722c43843310-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.703120 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.703060    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.703290 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.703258    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.703630 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.703387    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.703630 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.703437    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.703769 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.703669    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.705318 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.705288    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa360b6-12a9-4418-b3f2-722c43843310-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.705435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.705289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17345f0b-eb61-4bdf-9659-b0c78fb3967c-proxy-tls\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.709672 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.709648    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xsd\" (UniqueName: \"kubernetes.io/projected/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-api-access-z2xsd\") pod \"isvc-canary-create-v2-predictor-79c6bd57b5-4k96v\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.709793 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.709776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn7nv\" (UniqueName: \"kubernetes.io/projected/7aa360b6-12a9-4418-b3f2-722c43843310-kube-api-access-kn7nv\") pod \"isvc-canary-create-predictor-7d5b67cc88-5kpq9\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:44.777378 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.777348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:44.816418 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:44.816291    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:45.049464 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:45.049358    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"]
Sep 15 16:48:45.052010 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:48:45.051983    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17345f0b_eb61_4bdf_9659_b0c78fb3967c.slice/crio-4a5f4f0be071f78e90e5280d93d7be513d750e4e97e1cd534794e00cec7da81a WatchSource:0}: Error finding container 4a5f4f0be071f78e90e5280d93d7be513d750e4e97e1cd534794e00cec7da81a: Status 404 returned error can't find the container with id 4a5f4f0be071f78e90e5280d93d7be513d750e4e97e1cd534794e00cec7da81a
Sep 15 16:48:45.093560 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:45.093522    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"]
Sep 15 16:48:45.095436 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:48:45.095380    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa360b6_12a9_4418_b3f2_722c43843310.slice/crio-c83981bfab80ad096cb3e33f4683efa440fb93327d979c00cefadcaedeb9bdf6 WatchSource:0}: Error finding container c83981bfab80ad096cb3e33f4683efa440fb93327d979c00cefadcaedeb9bdf6: Status 404 returned error can't find the container with id c83981bfab80ad096cb3e33f4683efa440fb93327d979c00cefadcaedeb9bdf6
Sep 15 16:48:45.559964 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:45.559920    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:45.766815 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:45.766770    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" event={"ID":"7aa360b6-12a9-4418-b3f2-722c43843310","Type":"ContainerStarted","Data":"816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434"}
Sep 15 16:48:45.767013 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:45.766850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" event={"ID":"7aa360b6-12a9-4418-b3f2-722c43843310","Type":"ContainerStarted","Data":"c83981bfab80ad096cb3e33f4683efa440fb93327d979c00cefadcaedeb9bdf6"}
Sep 15 16:48:45.768463 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:45.768436    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" event={"ID":"17345f0b-eb61-4bdf-9659-b0c78fb3967c","Type":"ContainerStarted","Data":"d72ea4c6becb395a1e314a52ca70351528fa3865fcad736cb2317bdbcbb74c72"}
Sep 15 16:48:45.768588 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:45.768484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" event={"ID":"17345f0b-eb61-4bdf-9659-b0c78fb3967c","Type":"ContainerStarted","Data":"4a5f4f0be071f78e90e5280d93d7be513d750e4e97e1cd534794e00cec7da81a"}
Sep 15 16:48:49.785547 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:49.785515    2564 generic.go:356] "Generic (PLEG): container finished" podID="7aa360b6-12a9-4418-b3f2-722c43843310" containerID="816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434" exitCode=0
Sep 15 16:48:49.786009 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:49.785600    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" event={"ID":"7aa360b6-12a9-4418-b3f2-722c43843310","Type":"ContainerDied","Data":"816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434"}
Sep 15 16:48:49.787188 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:49.787169    2564 generic.go:356] "Generic (PLEG): container finished" podID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerID="d72ea4c6becb395a1e314a52ca70351528fa3865fcad736cb2317bdbcbb74c72" exitCode=0
Sep 15 16:48:49.787264 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:49.787223    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" event={"ID":"17345f0b-eb61-4bdf-9659-b0c78fb3967c","Type":"ContainerDied","Data":"d72ea4c6becb395a1e314a52ca70351528fa3865fcad736cb2317bdbcbb74c72"}
Sep 15 16:48:50.793554 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.793521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" event={"ID":"7aa360b6-12a9-4418-b3f2-722c43843310","Type":"ContainerStarted","Data":"13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7"}
Sep 15 16:48:50.793976 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.793590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" event={"ID":"7aa360b6-12a9-4418-b3f2-722c43843310","Type":"ContainerStarted","Data":"292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd"}
Sep 15 16:48:50.793976 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.793811    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:50.795591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.795566    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" event={"ID":"17345f0b-eb61-4bdf-9659-b0c78fb3967c","Type":"ContainerStarted","Data":"913a25b43aa07f682e75ed65a5dbdea85462f4be93e6d5186ff300334ae2418a"}
Sep 15 16:48:50.795721 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.795611    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" event={"ID":"17345f0b-eb61-4bdf-9659-b0c78fb3967c","Type":"ContainerStarted","Data":"8c96e05eb86b2f52ef44fdd77d24388433c094fe4c0f13b266f27308481e961f"}
Sep 15 16:48:50.795792 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.795777    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:50.795848 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.795819    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:50.797311 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.797279    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:48:50.811783 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.811736    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podStartSLOduration=3.058765046 podStartE2EDuration="6.811725439s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.752960393s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:48:50.810334833 +0000 UTC m=+1335.146289764" watchObservedRunningTime="2026-09-15 16:48:50.811725439 +0000 UTC m=+1335.147680369"
Sep 15 16:48:50.826811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:50.826742    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podStartSLOduration=2.9578352519999997 podStartE2EDuration="6.826729602s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.86889435s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:48:50.826216955 +0000 UTC m=+1335.162171887" watchObservedRunningTime="2026-09-15 16:48:50.826729602 +0000 UTC m=+1335.162684532"
Sep 15 16:48:51.800820 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:51.800773    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:48:51.800820 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:51.800807    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:51.802129 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:51.802103    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:48:52.804967 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:52.804926    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:48:55.559981 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:55.559937    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Sep 15 16:48:56.806163 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:56.806120    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:48:56.806761 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:56.806730    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:48:57.810516 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:57.810486    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:48:57.811239 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:48:57.811194    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:49:02.167614 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:02.167584    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:49:02.667805 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:02.667767    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"]
Sep 15 16:49:02.852801 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:02.852769    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-serving-cert: secret "raw-sklearn-runtime-d5318-predictor-serving-cert" not found
Sep 15 16:49:02.852963 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:02.852861    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls podName:1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3 nodeName:}" failed. No retries permitted until 2026-09-15 16:49:03.352835922 +0000 UTC m=+1347.688790844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls") pod "raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3") : secret "raw-sklearn-runtime-d5318-predictor-serving-cert" not found
Sep 15 16:49:03.356952 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:03.356918    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-serving-cert: secret "raw-sklearn-runtime-d5318-predictor-serving-cert" not found
Sep 15 16:49:03.357334 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:03.356989    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls podName:1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3 nodeName:}" failed. No retries permitted until 2026-09-15 16:49:04.356973556 +0000 UTC m=+1348.692928465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls") pod "raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3") : secret "raw-sklearn-runtime-d5318-predictor-serving-cert" not found
Sep 15 16:49:03.851217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:03.851176    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container" containerID="cri-o://cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09" gracePeriod=30
Sep 15 16:49:03.851217 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:03.851210    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kube-rbac-proxy" containerID="cri-o://40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465" gracePeriod=30
Sep 15 16:49:04.366681 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:04.366642    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-serving-cert: secret "raw-sklearn-runtime-d5318-predictor-serving-cert" not found
Sep 15 16:49:04.367043 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:04.366719    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls podName:1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3 nodeName:}" failed. No retries permitted until 2026-09-15 16:49:06.366703681 +0000 UTC m=+1350.702658610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls") pod "raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3") : secret "raw-sklearn-runtime-d5318-predictor-serving-cert" not found
Sep 15 16:49:04.857450 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:04.857418    2564 generic.go:356] "Generic (PLEG): container finished" podID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerID="40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465" exitCode=2
Sep 15 16:49:04.857616 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:04.857425    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" event={"ID":"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3","Type":"ContainerDied","Data":"40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465"}
Sep 15 16:49:05.554362 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:05.554322    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Sep 15 16:49:06.383107 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:06.383076    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-serving-cert: secret "raw-sklearn-runtime-d5318-predictor-serving-cert" not found
Sep 15 16:49:06.383313 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:06.383146    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls podName:1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3 nodeName:}" failed. No retries permitted until 2026-09-15 16:49:10.383130844 +0000 UTC m=+1354.719085753 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls") pod "raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3") : secret "raw-sklearn-runtime-d5318-predictor-serving-cert" not found
Sep 15 16:49:06.807068 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:06.807036    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:49:07.811375 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:07.811340    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:49:08.594875 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.594847    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:49:08.701794 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.701702    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-rbac-proxy-sar-config\") pod \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") "
Sep 15 16:49:08.701794 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.701770    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls\") pod \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") "
Sep 15 16:49:08.702015 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.701827    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kserve-provision-location\") pod \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") "
Sep 15 16:49:08.702015 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.701852    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tpb84\" (UniqueName: \"kubernetes.io/projected/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-api-access-tpb84\") pod \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") "
Sep 15 16:49:08.702015 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.701875    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-cabundle-cert\") pod \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\" (UID: \"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3\") "
Sep 15 16:49:08.702180 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.702151    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:49:08.702249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.702184    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:49:08.702302 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.702251    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:49:08.704156 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.704134    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:49:08.704156 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.704136    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-api-access-tpb84" (OuterVolumeSpecName: "kube-api-access-tpb84") pod "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" (UID: "1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3"). InnerVolumeSpecName "kube-api-access-tpb84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:49:08.802750 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.802702    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:49:08.802750 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.802744    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:49:08.802750 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.802754    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:49:08.802750 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.802764    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpb84\" (UniqueName: \"kubernetes.io/projected/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-kube-api-access-tpb84\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:49:08.803022 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.802772    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:49:08.874766 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.874731    2564 generic.go:356] "Generic (PLEG): container finished" podID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerID="cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09" exitCode=0
Sep 15 16:49:08.875148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.874792    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"
Sep 15 16:49:08.875148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.874807    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" event={"ID":"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3","Type":"ContainerDied","Data":"cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09"}
Sep 15 16:49:08.875148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.874874    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d" event={"ID":"1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3","Type":"ContainerDied","Data":"bd25d3ae7180066b40b6052767158b3d13251bc6fa2bc1653f5855ad869a9d21"}
Sep 15 16:49:08.875148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.874903    2564 scope.go:117] "RemoveContainer" containerID="40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465"
Sep 15 16:49:08.883596 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.883568    2564 scope.go:117] "RemoveContainer" containerID="cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09"
Sep 15 16:49:08.892655 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.892621    2564 scope.go:117] "RemoveContainer" containerID="73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499"
Sep 15 16:49:08.895827 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.895808    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"]
Sep 15 16:49:08.899255 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.899235    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-d5318-predictor-6bb6675c77-6ws7d"]
Sep 15 16:49:08.901047 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.901034    2564 scope.go:117] "RemoveContainer" containerID="40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465"
Sep 15 16:49:08.901312 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:08.901296    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465\": container with ID starting with 40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465 not found: ID does not exist" containerID="40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465"
Sep 15 16:49:08.901372 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.901320    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465"} err="failed to get container status \"40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465\": rpc error: code = NotFound desc = could not find container \"40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465\": container with ID starting with 40db4bd6ae353d5efe734a19e79e0f88cee24c5cf6c135c86b261b1532710465 not found: ID does not exist"
Sep 15 16:49:08.901372 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.901336    2564 scope.go:117] "RemoveContainer" containerID="cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09"
Sep 15 16:49:08.901600 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:08.901584    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09\": container with ID starting with cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09 not found: ID does not exist" containerID="cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09"
Sep 15 16:49:08.901644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.901604    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09"} err="failed to get container status \"cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09\": rpc error: code = NotFound desc = could not find container \"cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09\": container with ID starting with cdc27fb520f3a6f35025caa73f847b4597278a2e07134647ebf2ba5f4cb36f09 not found: ID does not exist"
Sep 15 16:49:08.901644 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.901617    2564 scope.go:117] "RemoveContainer" containerID="73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499"
Sep 15 16:49:08.901898 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:49:08.901877    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499\": container with ID starting with 73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499 not found: ID does not exist" containerID="73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499"
Sep 15 16:49:08.901952 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:08.901903    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499"} err="failed to get container status \"73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499\": rpc error: code = NotFound desc = could not find container \"73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499\": container with ID starting with 73effdf337d71c427dc80a975750a23093e6bb381122caa4f455a07c9d703499 not found: ID does not exist"
Sep 15 16:49:10.164972 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:10.164944    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" path="/var/lib/kubelet/pods/1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3/volumes"
Sep 15 16:49:16.807002 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:16.806957    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:49:17.811151 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:17.811114    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:49:26.807110 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:26.807071    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:49:27.812109 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:27.812072    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:49:36.807536 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:36.807498    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:49:37.812019 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:37.811979    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:49:46.807056 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:46.807015    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:49:47.811598 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:47.811557    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:49:56.807365 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:56.807314    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Sep 15 16:49:57.811775 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:49:57.811733    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 15 16:50:04.163987 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:04.163959    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:50:06.807951 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:06.807924    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:50:06.934633 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:06.934605    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"]
Sep 15 16:50:06.934841 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:06.934815    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container" containerID="cri-o://292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd" gracePeriod=30
Sep 15 16:50:06.934895 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:06.934850    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kube-rbac-proxy" containerID="cri-o://13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7" gracePeriod=30
Sep 15 16:50:07.107375 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:07.107288    2564 generic.go:356] "Generic (PLEG): container finished" podID="7aa360b6-12a9-4418-b3f2-722c43843310" containerID="13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7" exitCode=2
Sep 15 16:50:07.107540 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:07.107371    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" event={"ID":"7aa360b6-12a9-4418-b3f2-722c43843310","Type":"ContainerDied","Data":"13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7"}
Sep 15 16:50:07.805387 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:07.805347    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 15 16:50:09.555579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:09.555540    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"]
Sep 15 16:50:09.556030 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:09.555951    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container" containerID="cri-o://8c96e05eb86b2f52ef44fdd77d24388433c094fe4c0f13b266f27308481e961f" gracePeriod=30
Sep 15 16:50:09.556256 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:09.556206    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kube-rbac-proxy" containerID="cri-o://913a25b43aa07f682e75ed65a5dbdea85462f4be93e6d5186ff300334ae2418a" gracePeriod=30
Sep 15 16:50:10.121226 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:10.121193    2564 generic.go:356] "Generic (PLEG): container finished" podID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerID="913a25b43aa07f682e75ed65a5dbdea85462f4be93e6d5186ff300334ae2418a" exitCode=2
Sep 15 16:50:10.121435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:10.121251    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" event={"ID":"17345f0b-eb61-4bdf-9659-b0c78fb3967c","Type":"ContainerDied","Data":"913a25b43aa07f682e75ed65a5dbdea85462f4be93e6d5186ff300334ae2418a"}
Sep 15 16:50:11.578447 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.578423    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:50:11.644007 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.643899    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn7nv\" (UniqueName: \"kubernetes.io/projected/7aa360b6-12a9-4418-b3f2-722c43843310-kube-api-access-kn7nv\") pod \"7aa360b6-12a9-4418-b3f2-722c43843310\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") "
Sep 15 16:50:11.644007 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.643953    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-cabundle-cert\") pod \"7aa360b6-12a9-4418-b3f2-722c43843310\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") "
Sep 15 16:50:11.644007 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.644000    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa360b6-12a9-4418-b3f2-722c43843310-kserve-provision-location\") pod \"7aa360b6-12a9-4418-b3f2-722c43843310\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") "
Sep 15 16:50:11.644285 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.644135    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa360b6-12a9-4418-b3f2-722c43843310-proxy-tls\") pod \"7aa360b6-12a9-4418-b3f2-722c43843310\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") "
Sep 15 16:50:11.644285 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.644204    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-kube-rbac-proxy-sar-config\") pod \"7aa360b6-12a9-4418-b3f2-722c43843310\" (UID: \"7aa360b6-12a9-4418-b3f2-722c43843310\") "
Sep 15 16:50:11.644425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.644292    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aa360b6-12a9-4418-b3f2-722c43843310-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7aa360b6-12a9-4418-b3f2-722c43843310" (UID: "7aa360b6-12a9-4418-b3f2-722c43843310"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:50:11.644425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.644314    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7aa360b6-12a9-4418-b3f2-722c43843310" (UID: "7aa360b6-12a9-4418-b3f2-722c43843310"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:50:11.644536 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.644469    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7aa360b6-12a9-4418-b3f2-722c43843310-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:11.644536 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.644481    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:11.644638 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.644614    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7aa360b6-12a9-4418-b3f2-722c43843310" (UID: "7aa360b6-12a9-4418-b3f2-722c43843310"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:50:11.646239 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.646213    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa360b6-12a9-4418-b3f2-722c43843310-kube-api-access-kn7nv" (OuterVolumeSpecName: "kube-api-access-kn7nv") pod "7aa360b6-12a9-4418-b3f2-722c43843310" (UID: "7aa360b6-12a9-4418-b3f2-722c43843310"). InnerVolumeSpecName "kube-api-access-kn7nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:50:11.646388 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.646244    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa360b6-12a9-4418-b3f2-722c43843310-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7aa360b6-12a9-4418-b3f2-722c43843310" (UID: "7aa360b6-12a9-4418-b3f2-722c43843310"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:50:11.745028 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.744991    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn7nv\" (UniqueName: \"kubernetes.io/projected/7aa360b6-12a9-4418-b3f2-722c43843310-kube-api-access-kn7nv\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:11.745028 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.745019    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa360b6-12a9-4418-b3f2-722c43843310-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:11.745028 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.745029    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7aa360b6-12a9-4418-b3f2-722c43843310-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:11.801288 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:11.801245    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 15 16:50:12.131013 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.130978    2564 generic.go:356] "Generic (PLEG): container finished" podID="7aa360b6-12a9-4418-b3f2-722c43843310" containerID="292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd" exitCode=0
Sep 15 16:50:12.131207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.131035    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" event={"ID":"7aa360b6-12a9-4418-b3f2-722c43843310","Type":"ContainerDied","Data":"292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd"}
Sep 15 16:50:12.131207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.131085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9" event={"ID":"7aa360b6-12a9-4418-b3f2-722c43843310","Type":"ContainerDied","Data":"c83981bfab80ad096cb3e33f4683efa440fb93327d979c00cefadcaedeb9bdf6"}
Sep 15 16:50:12.131207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.131110    2564 scope.go:117] "RemoveContainer" containerID="13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7"
Sep 15 16:50:12.131207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.131037    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"
Sep 15 16:50:12.141175 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.140813    2564 scope.go:117] "RemoveContainer" containerID="292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd"
Sep 15 16:50:12.148669 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.148651    2564 scope.go:117] "RemoveContainer" containerID="816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434"
Sep 15 16:50:12.156103 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.156083    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"]
Sep 15 16:50:12.156252 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.156246    2564 scope.go:117] "RemoveContainer" containerID="13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7"
Sep 15 16:50:12.156602 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:50:12.156586    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7\": container with ID starting with 13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7 not found: ID does not exist" containerID="13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7"
Sep 15 16:50:12.156673 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.156610    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7"} err="failed to get container status \"13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7\": rpc error: code = NotFound desc = could not find container \"13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7\": container with ID starting with 13b77fc7538ae987ee37a0448524bbe7018ad319a81311720547b296efe031f7 not found: ID does not exist"
Sep 15 16:50:12.156673 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.156625    2564 scope.go:117] "RemoveContainer" containerID="292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd"
Sep 15 16:50:12.156838 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:50:12.156820    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd\": container with ID starting with 292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd not found: ID does not exist" containerID="292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd"
Sep 15 16:50:12.156907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.156850    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd"} err="failed to get container status \"292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd\": rpc error: code = NotFound desc = could not find container \"292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd\": container with ID starting with 292d8879d9a4040a6f3f0b9af60e158e1f83d73d115c7f6e4873ee5293eb20dd not found: ID does not exist"
Sep 15 16:50:12.156907 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.156872    2564 scope.go:117] "RemoveContainer" containerID="816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434"
Sep 15 16:50:12.157119 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:50:12.157102    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434\": container with ID starting with 816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434 not found: ID does not exist" containerID="816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434"
Sep 15 16:50:12.157170 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.157122    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434"} err="failed to get container status \"816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434\": rpc error: code = NotFound desc = could not find container \"816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434\": container with ID starting with 816e0c967bfcc5b9881da5a0c196c2dd45faad174a4ba2c5381c794a65a89434 not found: ID does not exist"
Sep 15 16:50:12.163878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:12.163861    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-5kpq9"]
Sep 15 16:50:14.143512 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.143473    2564 generic.go:356] "Generic (PLEG): container finished" podID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerID="8c96e05eb86b2f52ef44fdd77d24388433c094fe4c0f13b266f27308481e961f" exitCode=0
Sep 15 16:50:14.143845 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.143522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" event={"ID":"17345f0b-eb61-4bdf-9659-b0c78fb3967c","Type":"ContainerDied","Data":"8c96e05eb86b2f52ef44fdd77d24388433c094fe4c0f13b266f27308481e961f"}
Sep 15 16:50:14.164758 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.164729    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" path="/var/lib/kubelet/pods/7aa360b6-12a9-4418-b3f2-722c43843310/volumes"
Sep 15 16:50:14.196151 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.196127    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:50:14.268321 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.268291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17345f0b-eb61-4bdf-9659-b0c78fb3967c-proxy-tls\") pod \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") "
Sep 15 16:50:14.268497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.268335    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kserve-provision-location\") pod \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") "
Sep 15 16:50:14.268497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.268353    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-rbac-proxy-sar-config\") pod \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") "
Sep 15 16:50:14.268497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.268435    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2xsd\" (UniqueName: \"kubernetes.io/projected/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-api-access-z2xsd\") pod \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") "
Sep 15 16:50:14.268497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.268456    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-cabundle-cert\") pod \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\" (UID: \"17345f0b-eb61-4bdf-9659-b0c78fb3967c\") "
Sep 15 16:50:14.268708 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.268671    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "17345f0b-eb61-4bdf-9659-b0c78fb3967c" (UID: "17345f0b-eb61-4bdf-9659-b0c78fb3967c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:50:14.268772 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.268750    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "17345f0b-eb61-4bdf-9659-b0c78fb3967c" (UID: "17345f0b-eb61-4bdf-9659-b0c78fb3967c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:50:14.268883 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.268861    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "17345f0b-eb61-4bdf-9659-b0c78fb3967c" (UID: "17345f0b-eb61-4bdf-9659-b0c78fb3967c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:50:14.270557 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.270534    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17345f0b-eb61-4bdf-9659-b0c78fb3967c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "17345f0b-eb61-4bdf-9659-b0c78fb3967c" (UID: "17345f0b-eb61-4bdf-9659-b0c78fb3967c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:50:14.270752 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.270729    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-api-access-z2xsd" (OuterVolumeSpecName: "kube-api-access-z2xsd") pod "17345f0b-eb61-4bdf-9659-b0c78fb3967c" (UID: "17345f0b-eb61-4bdf-9659-b0c78fb3967c"). InnerVolumeSpecName "kube-api-access-z2xsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:50:14.369121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.369028    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17345f0b-eb61-4bdf-9659-b0c78fb3967c-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:14.369121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.369064    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:14.369121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.369078    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:14.369121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.369093    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2xsd\" (UniqueName: \"kubernetes.io/projected/17345f0b-eb61-4bdf-9659-b0c78fb3967c-kube-api-access-z2xsd\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:14.369121 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.369107    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17345f0b-eb61-4bdf-9659-b0c78fb3967c-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:50:14.743553 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743477    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"]
Sep 15 16:50:14.743806 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743792    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kube-rbac-proxy"
Sep 15 16:50:14.743860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743809    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kube-rbac-proxy"
Sep 15 16:50:14.743860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743824    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="storage-initializer"
Sep 15 16:50:14.743860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743829    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="storage-initializer"
Sep 15 16:50:14.743860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743836    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="storage-initializer"
Sep 15 16:50:14.743860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743841    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="storage-initializer"
Sep 15 16:50:14.743860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743848    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container"
Sep 15 16:50:14.743860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743852    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container"
Sep 15 16:50:14.743860 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743860    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743868    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743880    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kube-rbac-proxy"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743885    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kube-rbac-proxy"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743893    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="storage-initializer"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743898    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="storage-initializer"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743904    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kube-rbac-proxy"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743909    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kube-rbac-proxy"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743918    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743923    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743974    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kube-rbac-proxy"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743982    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kube-rbac-proxy"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743990    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kserve-container"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.743996    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa360b6-12a9-4418-b3f2-722c43843310" containerName="kserve-container"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.744000    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" containerName="kube-rbac-proxy"
Sep 15 16:50:14.744099 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.744007    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1497ac79-5ebf-4b1e-84f6-1e71c1d0aaf3" containerName="kserve-container"
Sep 15 16:50:14.747352 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.747334    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.749450 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.749432    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 15 16:50:14.749623 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.749609    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 15 16:50:14.757387 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.757363    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"]
Sep 15 16:50:14.805814 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.805779    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"]
Sep 15 16:50:14.809161 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.809143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.811168 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.811149    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 15 16:50:14.817968 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.817945    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"]
Sep 15 16:50:14.873668 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873635    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.873797 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxms\" (UniqueName: \"kubernetes.io/projected/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-api-access-4zxms\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.873797 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873703    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.873797 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515f0fcb-43a2-4407-96b9-31a2411ef927-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.873797 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.873949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.873949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873843    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.873949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873875    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.873949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873909    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.873949 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.873943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6l4\" (UniqueName: \"kubernetes.io/projected/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-api-access-jw6l4\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.974286 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.974286 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515f0fcb-43a2-4407-96b9-31a2411ef927-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.974549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974304    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.974549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.974549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.974549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.974549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974432    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.974549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6l4\" (UniqueName: \"kubernetes.io/projected/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-api-access-jw6l4\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.974549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974486    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.974549 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:50:14.974514    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 15 16:50:14.974940 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:50:14.974582    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 15 16:50:14.974940 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:50:14.974594    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls podName:515f0fcb-43a2-4407-96b9-31a2411ef927 nodeName:}" failed. No retries permitted until 2026-09-15 16:50:15.47457152 +0000 UTC m=+1419.810526445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls") pod "isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" (UID: "515f0fcb-43a2-4407-96b9-31a2411ef927") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 15 16:50:14.974940 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:50:14.974653    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls podName:ce6946a7-8ccf-40f2-93a6-417bfdd060c5 nodeName:}" failed. No retries permitted until 2026-09-15 16:50:15.474634771 +0000 UTC m=+1419.810589683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls") pod "isvc-canary-promote-predictor-6d6bc676fb-pkkd6" (UID: "ce6946a7-8ccf-40f2-93a6-417bfdd060c5") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 15 16:50:14.974940 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974517    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxms\" (UniqueName: \"kubernetes.io/projected/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-api-access-4zxms\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.974940 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.974940 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.974749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515f0fcb-43a2-4407-96b9-31a2411ef927-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.975138 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.975029    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.975138 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.975123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.975318 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.975300    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.975366 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.975312    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:14.983076 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.983057    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6l4\" (UniqueName: \"kubernetes.io/projected/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-api-access-jw6l4\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:14.983289 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:14.983268    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxms\" (UniqueName: \"kubernetes.io/projected/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-api-access-4zxms\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:15.149260 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.149227    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v" event={"ID":"17345f0b-eb61-4bdf-9659-b0c78fb3967c","Type":"ContainerDied","Data":"4a5f4f0be071f78e90e5280d93d7be513d750e4e97e1cd534794e00cec7da81a"}
Sep 15 16:50:15.149687 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.149270    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"
Sep 15 16:50:15.149687 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.149294    2564 scope.go:117] "RemoveContainer" containerID="913a25b43aa07f682e75ed65a5dbdea85462f4be93e6d5186ff300334ae2418a"
Sep 15 16:50:15.157954 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.157935    2564 scope.go:117] "RemoveContainer" containerID="8c96e05eb86b2f52ef44fdd77d24388433c094fe4c0f13b266f27308481e961f"
Sep 15 16:50:15.165265 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.165245    2564 scope.go:117] "RemoveContainer" containerID="d72ea4c6becb395a1e314a52ca70351528fa3865fcad736cb2317bdbcbb74c72"
Sep 15 16:50:15.172005 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.171982    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"]
Sep 15 16:50:15.177231 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.177211    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-79c6bd57b5-4k96v"]
Sep 15 16:50:15.479746 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.479660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:15.479746 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.479707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:15.482266 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.482241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-pkkd6\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:15.482362 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.482241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:15.658760 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.658708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:15.719789 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.719745    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:15.932437 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.932333    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"]
Sep 15 16:50:15.934453 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:50:15.934430    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515f0fcb_43a2_4407_96b9_31a2411ef927.slice/crio-f7f78833d46ff38ab754a87ba82059958f052d0f183991ee7940d8cec54fff38 WatchSource:0}: Error finding container f7f78833d46ff38ab754a87ba82059958f052d0f183991ee7940d8cec54fff38: Status 404 returned error can't find the container with id f7f78833d46ff38ab754a87ba82059958f052d0f183991ee7940d8cec54fff38
Sep 15 16:50:15.992454 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:15.992420    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"]
Sep 15 16:50:15.996180 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:50:15.996155    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6946a7_8ccf_40f2_93a6_417bfdd060c5.slice/crio-b72cba1a04ac8054669858f1c08f8f67c3cb10443e929be554129b5655db4c9f WatchSource:0}: Error finding container b72cba1a04ac8054669858f1c08f8f67c3cb10443e929be554129b5655db4c9f: Status 404 returned error can't find the container with id b72cba1a04ac8054669858f1c08f8f67c3cb10443e929be554129b5655db4c9f
Sep 15 16:50:16.157078 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:16.157043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" event={"ID":"515f0fcb-43a2-4407-96b9-31a2411ef927","Type":"ContainerStarted","Data":"1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c"}
Sep 15 16:50:16.157632 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:16.157112    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" event={"ID":"515f0fcb-43a2-4407-96b9-31a2411ef927","Type":"ContainerStarted","Data":"f7f78833d46ff38ab754a87ba82059958f052d0f183991ee7940d8cec54fff38"}
Sep 15 16:50:16.166108 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:16.166066    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17345f0b-eb61-4bdf-9659-b0c78fb3967c" path="/var/lib/kubelet/pods/17345f0b-eb61-4bdf-9659-b0c78fb3967c/volumes"
Sep 15 16:50:16.166578 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:16.166550    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" event={"ID":"ce6946a7-8ccf-40f2-93a6-417bfdd060c5","Type":"ContainerStarted","Data":"ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf"}
Sep 15 16:50:16.166710 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:16.166625    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" event={"ID":"ce6946a7-8ccf-40f2-93a6-417bfdd060c5","Type":"ContainerStarted","Data":"b72cba1a04ac8054669858f1c08f8f67c3cb10443e929be554129b5655db4c9f"}
Sep 15 16:50:20.178288 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:20.178251    2564 generic.go:356] "Generic (PLEG): container finished" podID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerID="1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c" exitCode=0
Sep 15 16:50:20.178704 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:20.178326    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" event={"ID":"515f0fcb-43a2-4407-96b9-31a2411ef927","Type":"ContainerDied","Data":"1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c"}
Sep 15 16:50:20.179915 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:20.179894    2564 generic.go:356] "Generic (PLEG): container finished" podID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerID="ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf" exitCode=0
Sep 15 16:50:20.180027 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:20.179973    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" event={"ID":"ce6946a7-8ccf-40f2-93a6-417bfdd060c5","Type":"ContainerDied","Data":"ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf"}
Sep 15 16:50:21.186729 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.186681    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" event={"ID":"515f0fcb-43a2-4407-96b9-31a2411ef927","Type":"ContainerStarted","Data":"b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a"}
Sep 15 16:50:21.187139 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.186754    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" event={"ID":"515f0fcb-43a2-4407-96b9-31a2411ef927","Type":"ContainerStarted","Data":"798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765"}
Sep 15 16:50:21.187139 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.187043    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:21.187139 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.187098    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:21.188450 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.188412    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:50:21.189299 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.189273    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" event={"ID":"ce6946a7-8ccf-40f2-93a6-417bfdd060c5","Type":"ContainerStarted","Data":"2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780"}
Sep 15 16:50:21.189382 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.189342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" event={"ID":"ce6946a7-8ccf-40f2-93a6-417bfdd060c5","Type":"ContainerStarted","Data":"0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30"}
Sep 15 16:50:21.189700 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.189680    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:21.189780 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.189733    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:21.190764 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.190740    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:50:21.205918 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.205872    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podStartSLOduration=3.457090334 podStartE2EDuration="7.205857424s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.74876709s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:50:21.20497298 +0000 UTC m=+1425.540927912" watchObservedRunningTime="2026-09-15 16:50:21.205857424 +0000 UTC m=+1425.541812356"
Sep 15 16:50:21.224111 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:21.224065    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podStartSLOduration=3.50593588 podStartE2EDuration="7.224048215s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.718112335s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:50:21.222004639 +0000 UTC m=+1425.557959572" watchObservedRunningTime="2026-09-15 16:50:21.224048215 +0000 UTC m=+1425.560003148"
Sep 15 16:50:22.193929 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:22.193889    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:50:22.194333 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:22.193951    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:50:27.199061 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:27.198975    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:50:27.200084 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:27.200049    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:50:27.200220 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:27.200192    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:50:27.200650 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:27.200623    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:50:37.200493 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:37.200449    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:50:37.200916 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:37.200586    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:50:47.200504 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:47.200459    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:50:47.200881 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:47.200586    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:50:57.200375 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:57.200333    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:50:57.200875 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:50:57.200679    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:51:07.200318 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:07.200281    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:51:07.200876 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:07.200572    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:51:17.200910 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:17.200862    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:51:17.201594 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:17.200862    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:51:27.200222 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:27.200179    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:51:27.200713 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:27.200596    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:51:29.159543 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:29.159504    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 15 16:51:37.201411 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:37.201369    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:51:37.312161 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:37.312128    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"]
Sep 15 16:51:37.312436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:37.312353    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container" containerID="cri-o://0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30" gracePeriod=30
Sep 15 16:51:37.312524 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:37.312431    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kube-rbac-proxy" containerID="cri-o://2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780" gracePeriod=30
Sep 15 16:51:37.313261 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:37.313245    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:51:37.490693 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:37.490613    2564 generic.go:356] "Generic (PLEG): container finished" podID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerID="2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780" exitCode=2
Sep 15 16:51:37.490827 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:37.490684    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" event={"ID":"ce6946a7-8ccf-40f2-93a6-417bfdd060c5","Type":"ContainerDied","Data":"2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780"}
Sep 15 16:51:41.961074 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.961051    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:51:41.990075 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.990049    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kserve-provision-location\") pod \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") "
Sep 15 16:51:41.990221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.990098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-cabundle-cert\") pod \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") "
Sep 15 16:51:41.990221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.990151    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-rbac-proxy-sar-config\") pod \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") "
Sep 15 16:51:41.990221 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.990197    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zxms\" (UniqueName: \"kubernetes.io/projected/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-api-access-4zxms\") pod \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") "
Sep 15 16:51:41.990384 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.990225    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls\") pod \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\" (UID: \"ce6946a7-8ccf-40f2-93a6-417bfdd060c5\") "
Sep 15 16:51:41.990497 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.990471    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce6946a7-8ccf-40f2-93a6-417bfdd060c5" (UID: "ce6946a7-8ccf-40f2-93a6-417bfdd060c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:51:41.990549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.990531    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce6946a7-8ccf-40f2-93a6-417bfdd060c5" (UID: "ce6946a7-8ccf-40f2-93a6-417bfdd060c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:51:41.990583 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.990527    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce6946a7-8ccf-40f2-93a6-417bfdd060c5" (UID: "ce6946a7-8ccf-40f2-93a6-417bfdd060c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:51:41.992463 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.992428    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-api-access-4zxms" (OuterVolumeSpecName: "kube-api-access-4zxms") pod "ce6946a7-8ccf-40f2-93a6-417bfdd060c5" (UID: "ce6946a7-8ccf-40f2-93a6-417bfdd060c5"). InnerVolumeSpecName "kube-api-access-4zxms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:51:41.992607 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:41.992588    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce6946a7-8ccf-40f2-93a6-417bfdd060c5" (UID: "ce6946a7-8ccf-40f2-93a6-417bfdd060c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:51:42.091368 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.091318    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:51:42.091368 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.091365    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:51:42.091368 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.091378    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zxms\" (UniqueName: \"kubernetes.io/projected/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kube-api-access-4zxms\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:51:42.091368 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.091388    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:51:42.091667 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.091418    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce6946a7-8ccf-40f2-93a6-417bfdd060c5-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:51:42.511632 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.511601    2564 generic.go:356] "Generic (PLEG): container finished" podID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerID="0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30" exitCode=0
Sep 15 16:51:42.511911 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.511656    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"
Sep 15 16:51:42.511911 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.511667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" event={"ID":"ce6946a7-8ccf-40f2-93a6-417bfdd060c5","Type":"ContainerDied","Data":"0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30"}
Sep 15 16:51:42.511911 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.511727    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6" event={"ID":"ce6946a7-8ccf-40f2-93a6-417bfdd060c5","Type":"ContainerDied","Data":"b72cba1a04ac8054669858f1c08f8f67c3cb10443e929be554129b5655db4c9f"}
Sep 15 16:51:42.511911 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.511756    2564 scope.go:117] "RemoveContainer" containerID="2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780"
Sep 15 16:51:42.519815 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.519797    2564 scope.go:117] "RemoveContainer" containerID="0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30"
Sep 15 16:51:42.526962 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.526940    2564 scope.go:117] "RemoveContainer" containerID="ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf"
Sep 15 16:51:42.530549 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.530530    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"]
Sep 15 16:51:42.535501 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.535484    2564 scope.go:117] "RemoveContainer" containerID="2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780"
Sep 15 16:51:42.535709 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.535692    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-pkkd6"]
Sep 15 16:51:42.535781 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:51:42.535765    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780\": container with ID starting with 2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780 not found: ID does not exist" containerID="2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780"
Sep 15 16:51:42.535831 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.535790    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780"} err="failed to get container status \"2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780\": rpc error: code = NotFound desc = could not find container \"2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780\": container with ID starting with 2c87696fb7547125543379720bb895b3fdecbaaa96f3b50848642b22fb8fa780 not found: ID does not exist"
Sep 15 16:51:42.535831 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.535816    2564 scope.go:117] "RemoveContainer" containerID="0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30"
Sep 15 16:51:42.536059 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:51:42.536044    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30\": container with ID starting with 0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30 not found: ID does not exist" containerID="0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30"
Sep 15 16:51:42.536103 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.536066    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30"} err="failed to get container status \"0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30\": rpc error: code = NotFound desc = could not find container \"0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30\": container with ID starting with 0881bdd7a67dd7a44aa1eccecfe4b3849daa59513c31f1cf7a9805a46f1bbd30 not found: ID does not exist"
Sep 15 16:51:42.536103 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.536081    2564 scope.go:117] "RemoveContainer" containerID="ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf"
Sep 15 16:51:42.536321 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:51:42.536303    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf\": container with ID starting with ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf not found: ID does not exist" containerID="ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf"
Sep 15 16:51:42.536364 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:42.536328    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf"} err="failed to get container status \"ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf\": rpc error: code = NotFound desc = could not find container \"ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf\": container with ID starting with ac9919360cab925c605f0df065d3ee7190865a1a66851be28685890e0117eecf not found: ID does not exist"
Sep 15 16:51:44.164649 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:51:44.164617    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" path="/var/lib/kubelet/pods/ce6946a7-8ccf-40f2-93a6-417bfdd060c5/volumes"
Sep 15 16:53:15.052298 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:15.052266    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"]
Sep 15 16:53:15.052868 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:15.052525    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" containerID="cri-o://798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765" gracePeriod=30
Sep 15 16:53:15.052868 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:15.052606    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kube-rbac-proxy" containerID="cri-o://b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a" gracePeriod=30
Sep 15 16:53:15.868442 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:15.868409    2564 generic.go:356] "Generic (PLEG): container finished" podID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerID="b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a" exitCode=2
Sep 15 16:53:15.868442 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:15.868445    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" event={"ID":"515f0fcb-43a2-4407-96b9-31a2411ef927","Type":"ContainerDied","Data":"b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a"}
Sep 15 16:53:17.194744 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:17.194696    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 15 16:53:17.201020 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:17.200991    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Sep 15 16:53:19.702672 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.702649    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:53:19.884280 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.884189    2564 generic.go:356] "Generic (PLEG): container finished" podID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerID="798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765" exitCode=0
Sep 15 16:53:19.884280 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.884228    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" event={"ID":"515f0fcb-43a2-4407-96b9-31a2411ef927","Type":"ContainerDied","Data":"798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765"}
Sep 15 16:53:19.884280 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.884249    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"
Sep 15 16:53:19.884583 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.884306    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh" event={"ID":"515f0fcb-43a2-4407-96b9-31a2411ef927","Type":"ContainerDied","Data":"f7f78833d46ff38ab754a87ba82059958f052d0f183991ee7940d8cec54fff38"}
Sep 15 16:53:19.884583 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.884338    2564 scope.go:117] "RemoveContainer" containerID="b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a"
Sep 15 16:53:19.887947 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.887924    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-cabundle-cert\") pod \"515f0fcb-43a2-4407-96b9-31a2411ef927\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") "
Sep 15 16:53:19.888049 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.887986    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515f0fcb-43a2-4407-96b9-31a2411ef927-kserve-provision-location\") pod \"515f0fcb-43a2-4407-96b9-31a2411ef927\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") "
Sep 15 16:53:19.888049 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.888040    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls\") pod \"515f0fcb-43a2-4407-96b9-31a2411ef927\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") "
Sep 15 16:53:19.888171 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.888073    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw6l4\" (UniqueName: \"kubernetes.io/projected/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-api-access-jw6l4\") pod \"515f0fcb-43a2-4407-96b9-31a2411ef927\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") "
Sep 15 16:53:19.888171 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.888100    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-rbac-proxy-sar-config\") pod \"515f0fcb-43a2-4407-96b9-31a2411ef927\" (UID: \"515f0fcb-43a2-4407-96b9-31a2411ef927\") "
Sep 15 16:53:19.888326 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.888302    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "515f0fcb-43a2-4407-96b9-31a2411ef927" (UID: "515f0fcb-43a2-4407-96b9-31a2411ef927"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:53:19.888326 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.888307    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/515f0fcb-43a2-4407-96b9-31a2411ef927-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "515f0fcb-43a2-4407-96b9-31a2411ef927" (UID: "515f0fcb-43a2-4407-96b9-31a2411ef927"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:53:19.888662 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.888635    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "515f0fcb-43a2-4407-96b9-31a2411ef927" (UID: "515f0fcb-43a2-4407-96b9-31a2411ef927"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:53:19.890364 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.890337    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "515f0fcb-43a2-4407-96b9-31a2411ef927" (UID: "515f0fcb-43a2-4407-96b9-31a2411ef927"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:53:19.890515 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.890489    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-api-access-jw6l4" (OuterVolumeSpecName: "kube-api-access-jw6l4") pod "515f0fcb-43a2-4407-96b9-31a2411ef927" (UID: "515f0fcb-43a2-4407-96b9-31a2411ef927"). InnerVolumeSpecName "kube-api-access-jw6l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:53:19.892687 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.892672    2564 scope.go:117] "RemoveContainer" containerID="798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765"
Sep 15 16:53:19.903088 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.903073    2564 scope.go:117] "RemoveContainer" containerID="1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c"
Sep 15 16:53:19.910035 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.910020    2564 scope.go:117] "RemoveContainer" containerID="b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a"
Sep 15 16:53:19.910299 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:53:19.910283    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a\": container with ID starting with b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a not found: ID does not exist" containerID="b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a"
Sep 15 16:53:19.910345 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.910307    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a"} err="failed to get container status \"b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a\": rpc error: code = NotFound desc = could not find container \"b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a\": container with ID starting with b84469b135299ce361267684950cdfa090e690e42bce633011de264704675e6a not found: ID does not exist"
Sep 15 16:53:19.910345 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.910324    2564 scope.go:117] "RemoveContainer" containerID="798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765"
Sep 15 16:53:19.910564 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:53:19.910548    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765\": container with ID starting with 798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765 not found: ID does not exist" containerID="798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765"
Sep 15 16:53:19.910646 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.910568    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765"} err="failed to get container status \"798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765\": rpc error: code = NotFound desc = could not find container \"798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765\": container with ID starting with 798ed734e54fd76b6cc5e2d09db342534dca18e1ff9c5d51f9271f78ff386765 not found: ID does not exist"
Sep 15 16:53:19.910646 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.910581    2564 scope.go:117] "RemoveContainer" containerID="1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c"
Sep 15 16:53:19.910799 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:53:19.910782    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c\": container with ID starting with 1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c not found: ID does not exist" containerID="1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c"
Sep 15 16:53:19.910837 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.910805    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c"} err="failed to get container status \"1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c\": rpc error: code = NotFound desc = could not find container \"1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c\": container with ID starting with 1650ae14086f2ed0cdde804e408fc12ecec614836474312ffbd7817cdbd95a6c not found: ID does not exist"
Sep 15 16:53:19.988967 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.988931    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:53:19.988967 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.988960    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/515f0fcb-43a2-4407-96b9-31a2411ef927-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:53:19.988967 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.988970    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/515f0fcb-43a2-4407-96b9-31a2411ef927-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:53:19.989188 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.988979    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6l4\" (UniqueName: \"kubernetes.io/projected/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-api-access-jw6l4\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:53:19.989188 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:19.988988    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/515f0fcb-43a2-4407-96b9-31a2411ef927-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:53:20.199542 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:20.199514    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"]
Sep 15 16:53:20.203109 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:20.203086    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-2wbrh"]
Sep 15 16:53:22.164826 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:22.164794    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" path="/var/lib/kubelet/pods/515f0fcb-43a2-4407-96b9-31a2411ef927/volumes"
Sep 15 16:53:29.134978 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.134902    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"]
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135182    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135193    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135202    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kube-rbac-proxy"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135208    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kube-rbac-proxy"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135215    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="storage-initializer"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135221    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="storage-initializer"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135234    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kube-rbac-proxy"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135239    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kube-rbac-proxy"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135248    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="storage-initializer"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135253    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="storage-initializer"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135262    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135267    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135309    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kserve-container"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135327    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kserve-container"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135334    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce6946a7-8ccf-40f2-93a6-417bfdd060c5" containerName="kube-rbac-proxy"
Sep 15 16:53:29.135336 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.135340    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="515f0fcb-43a2-4407-96b9-31a2411ef927" containerName="kube-rbac-proxy"
Sep 15 16:53:29.140083 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.140067    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.142375 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.142351    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 15 16:53:29.142517 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.142426    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 15 16:53:29.142575 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.142558    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:53:29.142621 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.142605    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lqw4s\""
Sep 15 16:53:29.143230 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.143210    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:53:29.148216 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.148195    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"]
Sep 15 16:53:29.163885 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.163863    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.164008 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.163896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwvg\" (UniqueName: \"kubernetes.io/projected/4bceb204-8a3f-41d6-a766-44e296a55209-kube-api-access-8pwvg\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.164069 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.164006    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bceb204-8a3f-41d6-a766-44e296a55209-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.265035 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.264997    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwvg\" (UniqueName: \"kubernetes.io/projected/4bceb204-8a3f-41d6-a766-44e296a55209-kube-api-access-8pwvg\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.265196 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.265098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bceb204-8a3f-41d6-a766-44e296a55209-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.265196 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.265136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.265343 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:53:29.265237    2564 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-serving-cert: secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 15 16:53:29.265343 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:53:29.265319    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls podName:4bceb204-8a3f-41d6-a766-44e296a55209 nodeName:}" failed. No retries permitted until 2026-09-15 16:53:29.765299647 +0000 UTC m=+1614.101254557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls") pod "message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" (UID: "4bceb204-8a3f-41d6-a766-44e296a55209") : secret "message-dumper-raw-cipn-predictor-serving-cert" not found
Sep 15 16:53:29.265837 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.265812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bceb204-8a3f-41d6-a766-44e296a55209-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.275662 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.275632    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwvg\" (UniqueName: \"kubernetes.io/projected/4bceb204-8a3f-41d6-a766-44e296a55209-kube-api-access-8pwvg\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.769816 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.769780    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:29.772299 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:29.772266    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-7vhqb\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:30.050526 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:30.050438    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:30.324472 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:30.324448    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"]
Sep 15 16:53:30.326508 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:53:30.326480    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bceb204_8a3f_41d6_a766_44e296a55209.slice/crio-500f85626d11dd25da31d7bbba5aaa96de4960009dcfb116036f8d52bdab5674 WatchSource:0}: Error finding container 500f85626d11dd25da31d7bbba5aaa96de4960009dcfb116036f8d52bdab5674: Status 404 returned error can't find the container with id 500f85626d11dd25da31d7bbba5aaa96de4960009dcfb116036f8d52bdab5674
Sep 15 16:53:30.328425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:30.328386    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:53:30.927886 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:30.927847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" event={"ID":"4bceb204-8a3f-41d6-a766-44e296a55209","Type":"ContainerStarted","Data":"500f85626d11dd25da31d7bbba5aaa96de4960009dcfb116036f8d52bdab5674"}
Sep 15 16:53:31.933872 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:31.933786    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" event={"ID":"4bceb204-8a3f-41d6-a766-44e296a55209","Type":"ContainerStarted","Data":"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573"}
Sep 15 16:53:31.934257 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:31.933889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" event={"ID":"4bceb204-8a3f-41d6-a766-44e296a55209","Type":"ContainerStarted","Data":"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d"}
Sep 15 16:53:31.934257 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:31.933966    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:31.951683 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:31.951636    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" podStartSLOduration=1.709678919 podStartE2EDuration="2.951621345s" totalImagesPullingTime="1.241942426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:53:31.949899541 +0000 UTC m=+1616.285854472" watchObservedRunningTime="2026-09-15 16:53:31.951621345 +0000 UTC m=+1616.287576275"
Sep 15 16:53:32.938485 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:32.938454    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:32.940100 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:32.940079    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:39.950784 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:39.950754    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:53:49.164043 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.164007    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"]
Sep 15 16:53:49.168704 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.168682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.170964 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.170941    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:53:49.171102 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.170947    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:53:49.171207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.171192    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 15 16:53:49.171249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.171195    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 15 16:53:49.179006 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.178964    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"]
Sep 15 16:53:49.205083 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.205048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlmkx\" (UniqueName: \"kubernetes.io/projected/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-api-access-mlmkx\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.205083 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.205084    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790c6fc7-8614-4d5d-8603-f55922e0ca57-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.205315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.205106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.205315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.205129    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790c6fc7-8614-4d5d-8603-f55922e0ca57-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.205315 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.205208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.306215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.306176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlmkx\" (UniqueName: \"kubernetes.io/projected/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-api-access-mlmkx\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.306215 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.306214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790c6fc7-8614-4d5d-8603-f55922e0ca57-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.306520 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.306240    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.306520 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.306260    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790c6fc7-8614-4d5d-8603-f55922e0ca57-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.306520 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.306381    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.306685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.306645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790c6fc7-8614-4d5d-8603-f55922e0ca57-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.307077 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.307049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.307183 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.307049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.308938 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.308913    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790c6fc7-8614-4d5d-8603-f55922e0ca57-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.316103 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.316079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlmkx\" (UniqueName: \"kubernetes.io/projected/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-api-access-mlmkx\") pod \"isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.480466 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.480344    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:49.755309 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:49.755285    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"]
Sep 15 16:53:49.757450 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:53:49.757423    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790c6fc7_8614_4d5d_8603_f55922e0ca57.slice/crio-8da5b05bc334004a62f31fc9f30d8ffb173f39dccfe92d86084b0f255425558f WatchSource:0}: Error finding container 8da5b05bc334004a62f31fc9f30d8ffb173f39dccfe92d86084b0f255425558f: Status 404 returned error can't find the container with id 8da5b05bc334004a62f31fc9f30d8ffb173f39dccfe92d86084b0f255425558f
Sep 15 16:53:50.004190 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:50.004148    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerStarted","Data":"3e3dd93a9d193044382f4689f8040fbe2a6c45d5200e66d1fcf637e6df925e34"}
Sep 15 16:53:50.004353 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:50.004213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerStarted","Data":"8da5b05bc334004a62f31fc9f30d8ffb173f39dccfe92d86084b0f255425558f"}
Sep 15 16:53:54.019798 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:54.019771    2564 generic.go:356] "Generic (PLEG): container finished" podID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerID="3e3dd93a9d193044382f4689f8040fbe2a6c45d5200e66d1fcf637e6df925e34" exitCode=0
Sep 15 16:53:54.020162 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:54.019806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerDied","Data":"3e3dd93a9d193044382f4689f8040fbe2a6c45d5200e66d1fcf637e6df925e34"}
Sep 15 16:53:55.026422 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:55.026372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerStarted","Data":"3cfa2c9e412b2a3aece0b63c91789889363ceb187b7e3e559240fb4c3a7a5d36"}
Sep 15 16:53:55.026878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:55.026460    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerStarted","Data":"7c2d3c758b5016a56aec384d9c6d1b427e834f206c077246744101b1ee3a4291"}
Sep 15 16:53:55.026878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:55.026487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerStarted","Data":"674e764a5b3d00ed20e64ffce184880dd9faa6e5fffa217c5d20a6d337c82ae2"}
Sep 15 16:53:55.026878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:55.026760    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:55.027002 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:55.026899    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:55.028169 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:55.028143    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:53:55.047685 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:55.047645    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podStartSLOduration=2.222961406 podStartE2EDuration="6.047631587s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824670181s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:53:55.046265036 +0000 UTC m=+1639.382219958" watchObservedRunningTime="2026-09-15 16:53:55.047631587 +0000 UTC m=+1639.383586518"
Sep 15 16:53:56.031337 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:56.031303    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:53:56.031835 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:56.031492    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:53:56.032631 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:56.032607    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:53:57.035653 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:57.035610    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:53:57.036055 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:53:57.035947    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:54:02.040249 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:02.040218    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:54:02.040833 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:02.040791    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:54:02.041194 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:02.041176    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:54:12.040782 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:12.040743    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:54:12.041313 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:12.041199    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:54:22.040925 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:22.040878    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:54:22.041529 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:22.041325    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:54:32.040937 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:32.040894    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:54:32.041451 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:32.041367    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:54:42.041653 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:42.041607    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:54:42.042152 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:42.042105    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:54:52.041588 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:52.041547    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:54:52.042135 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:54:52.042051    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:55:02.040964 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:02.040874    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:02.041463 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:02.041288    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:55:12.041574 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:12.041544    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:55:12.042062 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:12.041819    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:55:24.254534 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.254506    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-7vhqb_4bceb204-8a3f-41d6-a766-44e296a55209/kserve-container/0.log"
Sep 15 16:55:24.445524 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.445494    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"]
Sep 15 16:55:24.445761 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.445726    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" containerID="cri-o://674e764a5b3d00ed20e64ffce184880dd9faa6e5fffa217c5d20a6d337c82ae2" gracePeriod=30
Sep 15 16:55:24.445935 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.445755    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" containerID="cri-o://3cfa2c9e412b2a3aece0b63c91789889363ceb187b7e3e559240fb4c3a7a5d36" gracePeriod=30
Sep 15 16:55:24.445935 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.445789    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy" containerID="cri-o://7c2d3c758b5016a56aec384d9c6d1b427e834f206c077246744101b1ee3a4291" gracePeriod=30
Sep 15 16:55:24.467695 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.467668    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"]
Sep 15 16:55:24.467921 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.467882    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" containerName="kserve-container" containerID="cri-o://edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d" gracePeriod=30
Sep 15 16:55:24.467985 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.467963    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" containerName="kube-rbac-proxy" containerID="cri-o://8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573" gracePeriod=30
Sep 15 16:55:24.707477 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.707452    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:55:24.791456 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.791421    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls\") pod \"4bceb204-8a3f-41d6-a766-44e296a55209\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") "
Sep 15 16:55:24.791640 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.791464    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pwvg\" (UniqueName: \"kubernetes.io/projected/4bceb204-8a3f-41d6-a766-44e296a55209-kube-api-access-8pwvg\") pod \"4bceb204-8a3f-41d6-a766-44e296a55209\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") "
Sep 15 16:55:24.791640 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.791513    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bceb204-8a3f-41d6-a766-44e296a55209-kube-rbac-proxy-sar-config\") pod \"4bceb204-8a3f-41d6-a766-44e296a55209\" (UID: \"4bceb204-8a3f-41d6-a766-44e296a55209\") "
Sep 15 16:55:24.791916 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.791889    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bceb204-8a3f-41d6-a766-44e296a55209-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4bceb204-8a3f-41d6-a766-44e296a55209" (UID: "4bceb204-8a3f-41d6-a766-44e296a55209"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:55:24.793700 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.793675    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bceb204-8a3f-41d6-a766-44e296a55209" (UID: "4bceb204-8a3f-41d6-a766-44e296a55209"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:55:24.793789 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.793731    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bceb204-8a3f-41d6-a766-44e296a55209-kube-api-access-8pwvg" (OuterVolumeSpecName: "kube-api-access-8pwvg") pod "4bceb204-8a3f-41d6-a766-44e296a55209" (UID: "4bceb204-8a3f-41d6-a766-44e296a55209"). InnerVolumeSpecName "kube-api-access-8pwvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:55:24.892705 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.892619    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bceb204-8a3f-41d6-a766-44e296a55209-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:55:24.892705 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.892646    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwvg\" (UniqueName: \"kubernetes.io/projected/4bceb204-8a3f-41d6-a766-44e296a55209-kube-api-access-8pwvg\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:55:24.892705 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:24.892657    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4bceb204-8a3f-41d6-a766-44e296a55209-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:55:25.106110 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.106080    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sfncz/must-gather-4pkgn"]
Sep 15 16:55:25.106385 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.106373    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" containerName="kube-rbac-proxy"
Sep 15 16:55:25.106469 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.106386    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" containerName="kube-rbac-proxy"
Sep 15 16:55:25.106469 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.106422    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" containerName="kserve-container"
Sep 15 16:55:25.106469 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.106428    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" containerName="kserve-container"
Sep 15 16:55:25.106567 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.106478    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" containerName="kserve-container"
Sep 15 16:55:25.106567 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.106489    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" containerName="kube-rbac-proxy"
Sep 15 16:55:25.109530 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.109508    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:55:25.111717 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.111676    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sfncz\"/\"default-dockercfg-mgxxq\""
Sep 15 16:55:25.111878 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.111776    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sfncz\"/\"kube-root-ca.crt\""
Sep 15 16:55:25.111960 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.111917    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sfncz\"/\"openshift-service-ca.crt\""
Sep 15 16:55:25.118425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.118378    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfncz/must-gather-4pkgn"]
Sep 15 16:55:25.194873 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.194784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbdrn\" (UniqueName: \"kubernetes.io/projected/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-kube-api-access-tbdrn\") pod \"must-gather-4pkgn\" (UID: \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\") " pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:55:25.194873 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.194829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-must-gather-output\") pod \"must-gather-4pkgn\" (UID: \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\") " pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:55:25.296109 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.296069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdrn\" (UniqueName: \"kubernetes.io/projected/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-kube-api-access-tbdrn\") pod \"must-gather-4pkgn\" (UID: \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\") " pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:55:25.296499 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.296119    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-must-gather-output\") pod \"must-gather-4pkgn\" (UID: \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\") " pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:55:25.296499 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.296432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-must-gather-output\") pod \"must-gather-4pkgn\" (UID: \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\") " pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:55:25.304148 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.304128    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdrn\" (UniqueName: \"kubernetes.io/projected/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-kube-api-access-tbdrn\") pod \"must-gather-4pkgn\" (UID: \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\") " pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:55:25.386872 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.386833    2564 generic.go:356] "Generic (PLEG): container finished" podID="4bceb204-8a3f-41d6-a766-44e296a55209" containerID="8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573" exitCode=2
Sep 15 16:55:25.386872 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.386857    2564 generic.go:356] "Generic (PLEG): container finished" podID="4bceb204-8a3f-41d6-a766-44e296a55209" containerID="edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d" exitCode=2
Sep 15 16:55:25.387116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.386886    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" event={"ID":"4bceb204-8a3f-41d6-a766-44e296a55209","Type":"ContainerDied","Data":"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573"}
Sep 15 16:55:25.387116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.386906    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"
Sep 15 16:55:25.387116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.386948    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" event={"ID":"4bceb204-8a3f-41d6-a766-44e296a55209","Type":"ContainerDied","Data":"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d"}
Sep 15 16:55:25.387116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.386967    2564 scope.go:117] "RemoveContainer" containerID="8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573"
Sep 15 16:55:25.387116 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.386972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb" event={"ID":"4bceb204-8a3f-41d6-a766-44e296a55209","Type":"ContainerDied","Data":"500f85626d11dd25da31d7bbba5aaa96de4960009dcfb116036f8d52bdab5674"}
Sep 15 16:55:25.389794 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.389773    2564 generic.go:356] "Generic (PLEG): container finished" podID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerID="7c2d3c758b5016a56aec384d9c6d1b427e834f206c077246744101b1ee3a4291" exitCode=2
Sep 15 16:55:25.389931 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.389823    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerDied","Data":"7c2d3c758b5016a56aec384d9c6d1b427e834f206c077246744101b1ee3a4291"}
Sep 15 16:55:25.395959 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.395943    2564 scope.go:117] "RemoveContainer" containerID="edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d"
Sep 15 16:55:25.403297 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.403282    2564 scope.go:117] "RemoveContainer" containerID="8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573"
Sep 15 16:55:25.403599 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:55:25.403582    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573\": container with ID starting with 8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573 not found: ID does not exist" containerID="8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573"
Sep 15 16:55:25.403682 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.403610    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573"} err="failed to get container status \"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573\": rpc error: code = NotFound desc = could not find container \"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573\": container with ID starting with 8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573 not found: ID does not exist"
Sep 15 16:55:25.403682 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.403631    2564 scope.go:117] "RemoveContainer" containerID="edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d"
Sep 15 16:55:25.403874 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:55:25.403854    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d\": container with ID starting with edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d not found: ID does not exist" containerID="edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d"
Sep 15 16:55:25.403917 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.403880    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d"} err="failed to get container status \"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d\": rpc error: code = NotFound desc = could not find container \"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d\": container with ID starting with edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d not found: ID does not exist"
Sep 15 16:55:25.403917 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.403895    2564 scope.go:117] "RemoveContainer" containerID="8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573"
Sep 15 16:55:25.404149 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.404131    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573"} err="failed to get container status \"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573\": rpc error: code = NotFound desc = could not find container \"8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573\": container with ID starting with 8b5baebbd5c7d8d1abdf45344737644fa2570c9753e20479b504cc34fc9a6573 not found: ID does not exist"
Sep 15 16:55:25.404185 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.404152    2564 scope.go:117] "RemoveContainer" containerID="edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d"
Sep 15 16:55:25.404381 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.404365    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d"} err="failed to get container status \"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d\": rpc error: code = NotFound desc = could not find container \"edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d\": container with ID starting with edc31d7138e3a6377aa7b087263cc56e3934074193bb175655d65616f4e5bc0d not found: ID does not exist"
Sep 15 16:55:25.408582 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.408564    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"]
Sep 15 16:55:25.412205 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.412186    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-7vhqb"]
Sep 15 16:55:25.439323 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.439299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:55:25.706574 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:25.706498    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sfncz/must-gather-4pkgn"]
Sep 15 16:55:25.709603 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:55:25.709578    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f57c8de_9903_4eaa_b9c1_cbb6cdee5c1c.slice/crio-54ddd001785896b13cabc8e6b926639cce54f4cdab07b60c59f5efb597539828 WatchSource:0}: Error finding container 54ddd001785896b13cabc8e6b926639cce54f4cdab07b60c59f5efb597539828: Status 404 returned error can't find the container with id 54ddd001785896b13cabc8e6b926639cce54f4cdab07b60c59f5efb597539828
Sep 15 16:55:26.164696 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:26.164665    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bceb204-8a3f-41d6-a766-44e296a55209" path="/var/lib/kubelet/pods/4bceb204-8a3f-41d6-a766-44e296a55209/volumes"
Sep 15 16:55:26.398072 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:26.398033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfncz/must-gather-4pkgn" event={"ID":"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c","Type":"ContainerStarted","Data":"54ddd001785896b13cabc8e6b926639cce54f4cdab07b60c59f5efb597539828"}
Sep 15 16:55:27.035941 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:27.035891    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 15 16:55:30.422655 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:30.422621    2564 generic.go:356] "Generic (PLEG): container finished" podID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerID="674e764a5b3d00ed20e64ffce184880dd9faa6e5fffa217c5d20a6d337c82ae2" exitCode=0
Sep 15 16:55:30.423150 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:30.422691    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerDied","Data":"674e764a5b3d00ed20e64ffce184880dd9faa6e5fffa217c5d20a6d337c82ae2"}
Sep 15 16:55:32.036591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:32.036561    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 15 16:55:32.041555 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:32.041520    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:32.041876 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:32.041848    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:55:32.436003 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:32.435667    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfncz/must-gather-4pkgn" event={"ID":"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c","Type":"ContainerStarted","Data":"825a1d5a63a88fbd7fe98843103baa4cad1335b89dd8bc46e759c1b0d7d09088"}
Sep 15 16:55:32.436003 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:32.435723    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfncz/must-gather-4pkgn" event={"ID":"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c","Type":"ContainerStarted","Data":"be54acdd81b38f45123494175cfc50f4a16ca43a35fbca41027d679941ba08fb"}
Sep 15 16:55:32.452438 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:32.452370    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sfncz/must-gather-4pkgn" podStartSLOduration=1.254853971 podStartE2EDuration="7.452353795s" totalImagesPullingTime="6.197499824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:55:32.451574589 +0000 UTC m=+1736.787529521" watchObservedRunningTime="2026-09-15 16:55:32.452353795 +0000 UTC m=+1736.788308743"
Sep 15 16:55:37.036647 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:37.036608    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 15 16:55:37.037061 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:37.036785    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:55:40.789535 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:40.789509    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:40.813902 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:40.813875    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:40.823532 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:40.823508    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:40.833339 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:40.833320    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:41.620975 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:41.620933    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:41.632640 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:41.632618    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:41.643712 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:41.643691    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:41.654759 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:41.654739    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:42.036474 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:42.036431    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 15 16:55:42.041492 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:42.041463    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:42.041940 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:42.041909    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:55:42.459883 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:42.459818    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:42.474576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:42.474556    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:42.485164 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:42.485140    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:42.494522 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:42.494496    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:43.259141 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:43.259113    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:43.271863 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:43.271836    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:43.281957 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:43.281939    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:43.292230 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:43.292214    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:44.076261 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:44.076235    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:44.087207 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:44.087184    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:44.097265 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:44.097245    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:44.111043 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:44.111023    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:44.915000 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:44.914968    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:44.925620 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:44.925599    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:44.937834 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:44.937810    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:44.948076 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:44.948060    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:45.741487 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:45.741449    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:45.753409 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:45.753380    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:45.765641 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:45.765600    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:45.776219 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:45.776199    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:46.572343 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:46.572320    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:46.584386 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:46.584363    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:46.593929 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:46.593913    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:46.604164 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:46.604148    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:47.035857 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:47.035821    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 15 16:55:47.421773 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:47.421706    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:47.433193 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:47.433168    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:47.443065 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:47.443049    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:47.453579 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:47.453560    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:48.262891 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:48.262864    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:48.277080 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:48.277056    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:48.287008 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:48.286991    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:48.296942 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:48.296925    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:49.102811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:49.102784    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:49.113906 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:49.113880    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:49.123190 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:49.123170    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:49.133756 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:49.133737    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:49.921792 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:49.921766    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:49.932792 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:49.932770    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:49.942287 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:49.942267    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:49.952264 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:49.952241    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:50.765125 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:50.765091    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:50.777193 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:50.777164    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:50.787203 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:50.787174    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:50.797629 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:50.797577    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:51.648814 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:51.648791    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kserve-container/0.log"
Sep 15 16:55:51.661008 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:51.660987    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/kube-rbac-proxy/0.log"
Sep 15 16:55:51.671076 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:51.671055    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/agent/0.log"
Sep 15 16:55:51.680088 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:51.680070    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb_790c6fc7-8614-4d5d-8603-f55922e0ca57/storage-initializer/0.log"
Sep 15 16:55:52.035908 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:52.035867    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 15 16:55:52.041436 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:52.041353    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 15 16:55:52.041612 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:52.041540    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:55:52.041772 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:52.041747    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:55:52.041904 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:52.041889    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:55:54.540564 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.540530    2564 generic.go:356] "Generic (PLEG): container finished" podID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerID="3cfa2c9e412b2a3aece0b63c91789889363ceb187b7e3e559240fb4c3a7a5d36" exitCode=0
Sep 15 16:55:54.540954 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.540593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerDied","Data":"3cfa2c9e412b2a3aece0b63c91789889363ceb187b7e3e559240fb4c3a7a5d36"}
Sep 15 16:55:54.542443 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.542419    2564 generic.go:356] "Generic (PLEG): container finished" podID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerID="be54acdd81b38f45123494175cfc50f4a16ca43a35fbca41027d679941ba08fb" exitCode=0
Sep 15 16:55:54.542562 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.542422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sfncz/must-gather-4pkgn" event={"ID":"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c","Type":"ContainerDied","Data":"be54acdd81b38f45123494175cfc50f4a16ca43a35fbca41027d679941ba08fb"}
Sep 15 16:55:54.542806 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.542793    2564 scope.go:117] "RemoveContainer" containerID="be54acdd81b38f45123494175cfc50f4a16ca43a35fbca41027d679941ba08fb"
Sep 15 16:55:54.590689 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.590664    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:55:54.657569 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.657535    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-cabundle-cert\") pod \"790c6fc7-8614-4d5d-8603-f55922e0ca57\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") "
Sep 15 16:55:54.657739 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.657588    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790c6fc7-8614-4d5d-8603-f55922e0ca57-kserve-provision-location\") pod \"790c6fc7-8614-4d5d-8603-f55922e0ca57\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") "
Sep 15 16:55:54.657739 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.657643    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790c6fc7-8614-4d5d-8603-f55922e0ca57-proxy-tls\") pod \"790c6fc7-8614-4d5d-8603-f55922e0ca57\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") "
Sep 15 16:55:54.657739 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.657659    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-rbac-proxy-sar-config\") pod \"790c6fc7-8614-4d5d-8603-f55922e0ca57\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") "
Sep 15 16:55:54.657739 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.657689    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlmkx\" (UniqueName: \"kubernetes.io/projected/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-api-access-mlmkx\") pod \"790c6fc7-8614-4d5d-8603-f55922e0ca57\" (UID: \"790c6fc7-8614-4d5d-8603-f55922e0ca57\") "
Sep 15 16:55:54.657992 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.657916    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/790c6fc7-8614-4d5d-8603-f55922e0ca57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "790c6fc7-8614-4d5d-8603-f55922e0ca57" (UID: "790c6fc7-8614-4d5d-8603-f55922e0ca57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:55:54.658054 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.658009    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "790c6fc7-8614-4d5d-8603-f55922e0ca57" (UID: "790c6fc7-8614-4d5d-8603-f55922e0ca57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:55:54.658098 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.658044    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "790c6fc7-8614-4d5d-8603-f55922e0ca57" (UID: "790c6fc7-8614-4d5d-8603-f55922e0ca57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:55:54.659931 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.659905    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-api-access-mlmkx" (OuterVolumeSpecName: "kube-api-access-mlmkx") pod "790c6fc7-8614-4d5d-8603-f55922e0ca57" (UID: "790c6fc7-8614-4d5d-8603-f55922e0ca57"). InnerVolumeSpecName "kube-api-access-mlmkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:55:54.660044 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.659931    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790c6fc7-8614-4d5d-8603-f55922e0ca57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "790c6fc7-8614-4d5d-8603-f55922e0ca57" (UID: "790c6fc7-8614-4d5d-8603-f55922e0ca57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:55:54.758572 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.758542    2564 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/790c6fc7-8614-4d5d-8603-f55922e0ca57-proxy-tls\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:55:54.758572 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.758571    2564 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:55:54.758751 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.758582    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlmkx\" (UniqueName: \"kubernetes.io/projected/790c6fc7-8614-4d5d-8603-f55922e0ca57-kube-api-access-mlmkx\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:55:54.758751 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.758591    2564 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/790c6fc7-8614-4d5d-8603-f55922e0ca57-cabundle-cert\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:55:54.758751 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.758600    2564 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/790c6fc7-8614-4d5d-8603-f55922e0ca57-kserve-provision-location\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:55:54.952261 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:54.952228    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sfncz_must-gather-4pkgn_8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c/gather/0.log"
Sep 15 16:55:55.511271 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511244    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bwj65/must-gather-v27vw"]
Sep 15 16:55:55.511604 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511591    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="storage-initializer"
Sep 15 16:55:55.511658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511607    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="storage-initializer"
Sep 15 16:55:55.511658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511623    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent"
Sep 15 16:55:55.511658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511630    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent"
Sep 15 16:55:55.511658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511642    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container"
Sep 15 16:55:55.511658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511648    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container"
Sep 15 16:55:55.511658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511654    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy"
Sep 15 16:55:55.511658 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511660    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy"
Sep 15 16:55:55.511871 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511717    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="agent"
Sep 15 16:55:55.511871 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511728    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kserve-container"
Sep 15 16:55:55.511871 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.511733    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" containerName="kube-rbac-proxy"
Sep 15 16:55:55.514236 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.514218    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwj65/must-gather-v27vw"
Sep 15 16:55:55.516808 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.516778    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bwj65\"/\"openshift-service-ca.crt\""
Sep 15 16:55:55.516942 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.516839    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bwj65\"/\"kube-root-ca.crt\""
Sep 15 16:55:55.517033 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.517013    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bwj65\"/\"default-dockercfg-l5mqt\""
Sep 15 16:55:55.528725 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.528704    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bwj65/must-gather-v27vw"]
Sep 15 16:55:55.549070 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.549040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb" event={"ID":"790c6fc7-8614-4d5d-8603-f55922e0ca57","Type":"ContainerDied","Data":"8da5b05bc334004a62f31fc9f30d8ffb173f39dccfe92d86084b0f255425558f"}
Sep 15 16:55:55.549470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.549090    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"
Sep 15 16:55:55.549470 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.549105    2564 scope.go:117] "RemoveContainer" containerID="3cfa2c9e412b2a3aece0b63c91789889363ceb187b7e3e559240fb4c3a7a5d36"
Sep 15 16:55:55.557440 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.557389    2564 scope.go:117] "RemoveContainer" containerID="7c2d3c758b5016a56aec384d9c6d1b427e834f206c077246744101b1ee3a4291"
Sep 15 16:55:55.564736 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.564719    2564 scope.go:117] "RemoveContainer" containerID="674e764a5b3d00ed20e64ffce184880dd9faa6e5fffa217c5d20a6d337c82ae2"
Sep 15 16:55:55.565086 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.565063    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7jx\" (UniqueName: \"kubernetes.io/projected/7b4ed265-c10e-4084-b308-c221555ef191-kube-api-access-vj7jx\") pod \"must-gather-v27vw\" (UID: \"7b4ed265-c10e-4084-b308-c221555ef191\") " pod="openshift-must-gather-bwj65/must-gather-v27vw"
Sep 15 16:55:55.565138 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.565118    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b4ed265-c10e-4084-b308-c221555ef191-must-gather-output\") pod \"must-gather-v27vw\" (UID: \"7b4ed265-c10e-4084-b308-c221555ef191\") " pod="openshift-must-gather-bwj65/must-gather-v27vw"
Sep 15 16:55:55.570576 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.570558    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"]
Sep 15 16:55:55.573668 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.572984    2564 scope.go:117] "RemoveContainer" containerID="3e3dd93a9d193044382f4689f8040fbe2a6c45d5200e66d1fcf637e6df925e34"
Sep 15 16:55:55.575425 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.575383    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-77c69cff6d-nt5lb"]
Sep 15 16:55:55.666337 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.666308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7jx\" (UniqueName: \"kubernetes.io/projected/7b4ed265-c10e-4084-b308-c221555ef191-kube-api-access-vj7jx\") pod \"must-gather-v27vw\" (UID: \"7b4ed265-c10e-4084-b308-c221555ef191\") " pod="openshift-must-gather-bwj65/must-gather-v27vw"
Sep 15 16:55:55.666502 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.666354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b4ed265-c10e-4084-b308-c221555ef191-must-gather-output\") pod \"must-gather-v27vw\" (UID: \"7b4ed265-c10e-4084-b308-c221555ef191\") " pod="openshift-must-gather-bwj65/must-gather-v27vw"
Sep 15 16:55:55.666705 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.666688    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7b4ed265-c10e-4084-b308-c221555ef191-must-gather-output\") pod \"must-gather-v27vw\" (UID: \"7b4ed265-c10e-4084-b308-c221555ef191\") " pod="openshift-must-gather-bwj65/must-gather-v27vw"
Sep 15 16:55:55.674017 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.673990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7jx\" (UniqueName: \"kubernetes.io/projected/7b4ed265-c10e-4084-b308-c221555ef191-kube-api-access-vj7jx\") pod \"must-gather-v27vw\" (UID: \"7b4ed265-c10e-4084-b308-c221555ef191\") " pod="openshift-must-gather-bwj65/must-gather-v27vw"
Sep 15 16:55:55.823261 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:55.823184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwj65/must-gather-v27vw"
Sep 15 16:55:56.090080 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:56.090010    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bwj65/must-gather-v27vw"]
Sep 15 16:55:56.093535 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:55:56.093511    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b4ed265_c10e_4084_b308_c221555ef191.slice/crio-5e478ba47bbb0b4feb03bb97f508566c0d9a3e15c9b4017f2f4f35881dd178b2 WatchSource:0}: Error finding container 5e478ba47bbb0b4feb03bb97f508566c0d9a3e15c9b4017f2f4f35881dd178b2: Status 404 returned error can't find the container with id 5e478ba47bbb0b4feb03bb97f508566c0d9a3e15c9b4017f2f4f35881dd178b2
Sep 15 16:55:56.164264 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:56.164235    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="790c6fc7-8614-4d5d-8603-f55922e0ca57" path="/var/lib/kubelet/pods/790c6fc7-8614-4d5d-8603-f55922e0ca57/volumes"
Sep 15 16:55:56.553700 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:56.553656    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwj65/must-gather-v27vw" event={"ID":"7b4ed265-c10e-4084-b308-c221555ef191","Type":"ContainerStarted","Data":"5e478ba47bbb0b4feb03bb97f508566c0d9a3e15c9b4017f2f4f35881dd178b2"}
Sep 15 16:55:57.561633 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:57.561593    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwj65/must-gather-v27vw" event={"ID":"7b4ed265-c10e-4084-b308-c221555ef191","Type":"ContainerStarted","Data":"397bf3dc616384d44d001afa9a34c5e47e6c32f6eadda0923bf08fb807064265"}
Sep 15 16:55:57.562115 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:57.561647    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwj65/must-gather-v27vw" event={"ID":"7b4ed265-c10e-4084-b308-c221555ef191","Type":"ContainerStarted","Data":"bd83ed1ac26f0e0a9f1c7939de0be3da5fb8490393d531c6fe0f31051098b1ab"}
Sep 15 16:55:57.577666 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:57.577608    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bwj65/must-gather-v27vw" podStartSLOduration=1.647221158 podStartE2EDuration="2.577587541s" totalImagesPullingTime="930.366383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:55:57.577272671 +0000 UTC m=+1761.913227602" watchObservedRunningTime="2026-09-15 16:55:57.577587541 +0000 UTC m=+1761.913542472"
Sep 15 16:55:58.331422 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:58.331365    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5tx7l_0b1e7e97-0a0e-400a-ad6c-99bbd53dca7e/global-pull-secret-syncer/0.log"
Sep 15 16:55:58.494031 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:58.494001    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-58fbn_a041ecec-9cc4-47e6-92c2-6e2213e5353f/konnectivity-agent/0.log"
Sep 15 16:55:58.547049 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:55:58.547018    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-221.ec2.internal_7fe48e1c24e6156964623e1b724b0c0d/haproxy/0.log"
Sep 15 16:56:00.358430 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.357291    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-sfncz/must-gather-4pkgn"]
Sep 15 16:56:00.358430 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.357520    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-sfncz/must-gather-4pkgn" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerName="copy" containerID="cri-o://825a1d5a63a88fbd7fe98843103baa4cad1335b89dd8bc46e759c1b0d7d09088" gracePeriod=2
Sep 15 16:56:00.359717 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:56:00.359672    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4pkgn\" is forbidden: User \"system:node:ip-10-0-130-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sfncz\": no relationship found between node 'ip-10-0-130-221.ec2.internal' and this object" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:56:00.362114 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.362095    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-sfncz/must-gather-4pkgn"]
Sep 15 16:56:00.582416 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.581844    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sfncz_must-gather-4pkgn_8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c/copy/0.log"
Sep 15 16:56:00.585747 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.585473    2564 generic.go:356] "Generic (PLEG): container finished" podID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerID="825a1d5a63a88fbd7fe98843103baa4cad1335b89dd8bc46e759c1b0d7d09088" exitCode=143
Sep 15 16:56:00.754358 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.754333    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sfncz_must-gather-4pkgn_8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c/copy/0.log"
Sep 15 16:56:00.758001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.757967    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:56:00.763833 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:56:00.763591    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4pkgn\" is forbidden: User \"system:node:ip-10-0-130-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sfncz\": no relationship found between node 'ip-10-0-130-221.ec2.internal' and this object" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:56:00.830354 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.830308    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-must-gather-output\") pod \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\" (UID: \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\") "
Sep 15 16:56:00.830667 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.830451    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbdrn\" (UniqueName: \"kubernetes.io/projected/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-kube-api-access-tbdrn\") pod \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\" (UID: \"8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c\") "
Sep 15 16:56:00.834022 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.833992    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" (UID: "8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:56:00.841313 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.841289    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-kube-api-access-tbdrn" (OuterVolumeSpecName: "kube-api-access-tbdrn") pod "8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" (UID: "8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c"). InnerVolumeSpecName "kube-api-access-tbdrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:56:00.932190 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.932156    2564 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-must-gather-output\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:56:00.932435 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:00.932418    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbdrn\" (UniqueName: \"kubernetes.io/projected/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c-kube-api-access-tbdrn\") on node \"ip-10-0-130-221.ec2.internal\" DevicePath \"\""
Sep 15 16:56:01.593684 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:01.593659    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sfncz_must-gather-4pkgn_8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c/copy/0.log"
Sep 15 16:56:01.594829 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:01.594807    2564 scope.go:117] "RemoveContainer" containerID="825a1d5a63a88fbd7fe98843103baa4cad1335b89dd8bc46e759c1b0d7d09088"
Sep 15 16:56:01.595133 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:01.595118    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:56:01.605584 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:56:01.605553    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4pkgn\" is forbidden: User \"system:node:ip-10-0-130-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sfncz\": no relationship found between node 'ip-10-0-130-221.ec2.internal' and this object" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:56:01.626420 ip-10-0-130-221 kubenswrapper[2564]: E0915 16:56:01.617340    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-4pkgn\" is forbidden: User \"system:node:ip-10-0-130-221.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sfncz\": no relationship found between node 'ip-10-0-130-221.ec2.internal' and this object" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" pod="openshift-must-gather-sfncz/must-gather-4pkgn"
Sep 15 16:56:01.626420 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:01.625011    2564 scope.go:117] "RemoveContainer" containerID="be54acdd81b38f45123494175cfc50f4a16ca43a35fbca41027d679941ba08fb"
Sep 15 16:56:02.167739 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.167703    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" path="/var/lib/kubelet/pods/8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c/volumes"
Sep 15 16:56:02.710196 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.710157    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-p4kc4_cd210cff-3446-4ca9-98d0-b8b82b29ac4d/kube-state-metrics/0.log"
Sep 15 16:56:02.735298 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.735266    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-p4kc4_cd210cff-3446-4ca9-98d0-b8b82b29ac4d/kube-rbac-proxy-main/0.log"
Sep 15 16:56:02.760876 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.760839    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-p4kc4_cd210cff-3446-4ca9-98d0-b8b82b29ac4d/kube-rbac-proxy-self/0.log"
Sep 15 16:56:02.806119 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.806088    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fd897789f-hhsqg_8424952e-1a56-4c1e-b194-6804e409d172/metrics-server/0.log"
Sep 15 16:56:02.836314 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.836275    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-rjm4n_058af8b2-822d-4644-b052-f62913a80db7/monitoring-plugin/0.log"
Sep 15 16:56:02.870379 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.870298    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95pf2_baa822fb-e94e-40f3-967e-6e4e04251bee/node-exporter/0.log"
Sep 15 16:56:02.893363 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.893332    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95pf2_baa822fb-e94e-40f3-967e-6e4e04251bee/kube-rbac-proxy/0.log"
Sep 15 16:56:02.916727 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:02.916695    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95pf2_baa822fb-e94e-40f3-967e-6e4e04251bee/init-textfile/0.log"
Sep 15 16:56:03.229192 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.229115    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_061d28c3-a733-4c26-8c2a-477b9e6ff0cd/prometheus/0.log"
Sep 15 16:56:03.252525 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.252498    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_061d28c3-a733-4c26-8c2a-477b9e6ff0cd/config-reloader/0.log"
Sep 15 16:56:03.277956 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.277925    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_061d28c3-a733-4c26-8c2a-477b9e6ff0cd/thanos-sidecar/0.log"
Sep 15 16:56:03.302461 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.302358    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_061d28c3-a733-4c26-8c2a-477b9e6ff0cd/kube-rbac-proxy-web/0.log"
Sep 15 16:56:03.329686 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.329658    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_061d28c3-a733-4c26-8c2a-477b9e6ff0cd/kube-rbac-proxy/0.log"
Sep 15 16:56:03.354183 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.354154    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_061d28c3-a733-4c26-8c2a-477b9e6ff0cd/kube-rbac-proxy-thanos/0.log"
Sep 15 16:56:03.377565 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.377536    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_061d28c3-a733-4c26-8c2a-477b9e6ff0cd/init-config-reloader/0.log"
Sep 15 16:56:03.408623 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.408591    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-srqlf_64da0150-d777-4d4f-93d9-d10d5eae5c06/prometheus-operator/0.log"
Sep 15 16:56:03.430629 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.430599    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-srqlf_64da0150-d777-4d4f-93d9-d10d5eae5c06/kube-rbac-proxy/0.log"
Sep 15 16:56:03.460536 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:03.460505    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-qx2l4_9e2982d1-5e96-47f9-8941-dbf39b319934/prometheus-operator-admission-webhook/0.log"
Sep 15 16:56:06.004809 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.004781    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-vnlcg_ad3e84de-141d-4b70-aeb4-f6f5663dae39/download-server/0.log"
Sep 15 16:56:06.542020 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.541988    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"]
Sep 15 16:56:06.542380 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.542361    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerName="gather"
Sep 15 16:56:06.542380 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.542382    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerName="gather"
Sep 15 16:56:06.542550 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.542416    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerName="copy"
Sep 15 16:56:06.542550 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.542424    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerName="copy"
Sep 15 16:56:06.542550 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.542483    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerName="gather"
Sep 15 16:56:06.542550 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.542493    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f57c8de-9903-4eaa-b9c1-cbb6cdee5c1c" containerName="copy"
Sep 15 16:56:06.545643 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.545620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.555818 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.555709    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"]
Sep 15 16:56:06.684171 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.684130    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-podres\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.684334 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.684210    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-lib-modules\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.684334 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.684284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxtk\" (UniqueName: \"kubernetes.io/projected/6b26c704-3008-4f46-801d-69acd8428fb7-kube-api-access-flxtk\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.684438 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.684333    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-proc\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.684476 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.684432    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-sys\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.785710 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.785670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxtk\" (UniqueName: \"kubernetes.io/projected/6b26c704-3008-4f46-801d-69acd8428fb7-kube-api-access-flxtk\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.785919 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.785740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-proc\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.785919 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.785778    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-sys\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.785919 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.785816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-podres\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.785919 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.785870    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-lib-modules\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.785919 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.785890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-proc\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.786227 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.785973    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-sys\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.786227 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.786026    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-lib-modules\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.786227 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.786084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b26c704-3008-4f46-801d-69acd8428fb7-podres\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.794292 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.794215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxtk\" (UniqueName: \"kubernetes.io/projected/6b26c704-3008-4f46-801d-69acd8428fb7-kube-api-access-flxtk\") pod \"perf-node-gather-daemonset-gf59c\" (UID: \"6b26c704-3008-4f46-801d-69acd8428fb7\") " pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:06.861861 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:06.861821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:07.155193 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:07.155123    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"]
Sep 15 16:56:07.158652 ip-10-0-130-221 kubenswrapper[2564]: W0915 16:56:07.158622    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b26c704_3008_4f46_801d_69acd8428fb7.slice/crio-1e1ca8ba3f76830d9c11eab21ddba6f9779f10f0875a1c4b02304fe073150933 WatchSource:0}: Error finding container 1e1ca8ba3f76830d9c11eab21ddba6f9779f10f0875a1c4b02304fe073150933: Status 404 returned error can't find the container with id 1e1ca8ba3f76830d9c11eab21ddba6f9779f10f0875a1c4b02304fe073150933
Sep 15 16:56:07.493276 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:07.493205    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wsm9l_326e7ec6-2ae7-4b30-b424-674d54fd4fce/dns/0.log"
Sep 15 16:56:07.516781 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:07.516754    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wsm9l_326e7ec6-2ae7-4b30-b424-674d54fd4fce/kube-rbac-proxy/0.log"
Sep 15 16:56:07.585361 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:07.585326    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rkjh7_3bd0a3f9-d122-4b8b-a617-38354c3e1687/dns-node-resolver/0.log"
Sep 15 16:56:07.641628 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:07.641589    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c" event={"ID":"6b26c704-3008-4f46-801d-69acd8428fb7","Type":"ContainerStarted","Data":"0e5d54f671547b7a4f76de094e1b288eb3ec6848246d31ac537e160e6907f3ec"}
Sep 15 16:56:07.641832 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:07.641650    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c" event={"ID":"6b26c704-3008-4f46-801d-69acd8428fb7","Type":"ContainerStarted","Data":"1e1ca8ba3f76830d9c11eab21ddba6f9779f10f0875a1c4b02304fe073150933"}
Sep 15 16:56:07.641832 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:07.641681    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:07.658656 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:07.658598    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c" podStartSLOduration=1.6585775699999998 podStartE2EDuration="1.65857757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:56:07.656880773 +0000 UTC m=+1771.992835702" watchObservedRunningTime="2026-09-15 16:56:07.65857757 +0000 UTC m=+1771.994532502"
Sep 15 16:56:08.085095 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:08.085067    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s2cwp_47f4965d-054d-40a3-95d7-5771be4ae2e9/node-ca/0.log"
Sep 15 16:56:09.164706 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:09.164669    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-prb6c_51e32a97-593e-4eee-bca2-92e61c8dedf4/serve-healthcheck-canary/0.log"
Sep 15 16:56:09.620125 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:09.620069    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8jm8c_ff240207-d755-42a9-9d1f-22f356486529/kube-rbac-proxy/0.log"
Sep 15 16:56:09.640910 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:09.640888    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8jm8c_ff240207-d755-42a9-9d1f-22f356486529/exporter/0.log"
Sep 15 16:56:09.664528 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:09.664507    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8jm8c_ff240207-d755-42a9-9d1f-22f356486529/extractor/0.log"
Sep 15 16:56:11.941876 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:11.941847    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-wfvf8_97fef691-d4b9-42b9-87f0-46d131ee1fc3/server/0.log"
Sep 15 16:56:12.076416 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:12.076368    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-xbhcn_da96b0c7-4cee-4070-8524-09a675071919/manager/0.log"
Sep 15 16:56:12.106703 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:12.106680    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5ztmz_db490de3-5409-4751-a244-8e0e00862792/s3-init/0.log"
Sep 15 16:56:12.122521 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:12.122497    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5ztmz_db490de3-5409-4751-a244-8e0e00862792/download-hf-model/0.log"
Sep 15 16:56:12.149431 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:12.149408    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-grwnl_9efc5be1-3a83-4513-a2e5-9441a6e48a65/seaweedfs/0.log"
Sep 15 16:56:13.658188 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:13.658162    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bwj65/perf-node-gather-daemonset-gf59c"
Sep 15 16:56:17.612172 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:17.612141    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h4rqc_b84e787d-7468-4967-963a-bdf15591987a/kube-multus-additional-cni-plugins/0.log"
Sep 15 16:56:17.633001 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:17.632970    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h4rqc_b84e787d-7468-4967-963a-bdf15591987a/egress-router-binary-copy/0.log"
Sep 15 16:56:17.653558 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:17.653534    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h4rqc_b84e787d-7468-4967-963a-bdf15591987a/cni-plugins/0.log"
Sep 15 16:56:17.674811 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:17.674740    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h4rqc_b84e787d-7468-4967-963a-bdf15591987a/bond-cni-plugin/0.log"
Sep 15 16:56:17.696866 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:17.696843    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h4rqc_b84e787d-7468-4967-963a-bdf15591987a/routeoverride-cni/0.log"
Sep 15 16:56:17.718102 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:17.718066    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h4rqc_b84e787d-7468-4967-963a-bdf15591987a/whereabouts-cni-bincopy/0.log"
Sep 15 16:56:17.741456 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:17.741433    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h4rqc_b84e787d-7468-4967-963a-bdf15591987a/whereabouts-cni/0.log"
Sep 15 16:56:18.062868 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:18.062845    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-p2j6z_0ac161c6-7af9-4c0f-9697-2ae5a38b377f/kube-multus/0.log"
Sep 15 16:56:18.095310 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:18.095292    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gt76w_3209aee8-cc28-4cc4-a3f0-c6af9ede9a60/network-metrics-daemon/0.log"
Sep 15 16:56:18.130320 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:18.130294    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gt76w_3209aee8-cc28-4cc4-a3f0-c6af9ede9a60/kube-rbac-proxy/0.log"
Sep 15 16:56:19.498160 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:19.498132    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcbrc_7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35/ovn-controller/0.log"
Sep 15 16:56:19.515309 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:19.515291    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcbrc_7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35/ovn-acl-logging/0.log"
Sep 15 16:56:19.536355 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:19.536334    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcbrc_7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35/kube-rbac-proxy-node/0.log"
Sep 15 16:56:19.557930 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:19.557909    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcbrc_7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35/kube-rbac-proxy-ovn-metrics/0.log"
Sep 15 16:56:19.577188 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:19.577167    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcbrc_7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35/northd/0.log"
Sep 15 16:56:19.597409 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:19.597381    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcbrc_7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35/nbdb/0.log"
Sep 15 16:56:19.620731 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:19.620711    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcbrc_7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35/sbdb/0.log"
Sep 15 16:56:19.664443 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:19.664415    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcbrc_7f127dd2-3cd7-4868-b39a-c9aeaaa2ac35/ovnkube-controller/0.log"
Sep 15 16:56:20.678633 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:20.678602    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-db66d_584a2e9a-89de-477a-9450-abdbc0d5b622/network-check-target-container/0.log"
Sep 15 16:56:21.553591 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:21.553567    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8kqvx_21f87036-4cef-43d9-b366-cecbb41ea28f/iptables-alerter/0.log"
Sep 15 16:56:22.190536 ip-10-0-130-221 kubenswrapper[2564]: I0915 16:56:22.190508    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xtg5k_61a16b0b-3551-4b57-a1b5-116f6b194072/tuned/0.log"