Sep 17 23:09:21.835653 ip-10-0-142-246 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 23:09:21.835683 ip-10-0-142-246 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 23:09:21.835697 ip-10-0-142-246 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 23:09:21.836026 ip-10-0-142-246 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 23:09:31.545012 ip-10-0-142-246 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 17 23:09:31.545027 ip-10-0-142-246 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 90ee8420842c477bbb974ee1ce341cc2 --
Sep 17 23:11:15.434584 ip-10-0-142-246 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:11:15.854225 ip-10-0-142-246 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:11:15.854225 ip-10-0-142-246 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:11:15.854225 ip-10-0-142-246 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:11:15.854225 ip-10-0-142-246 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:11:15.854225 ip-10-0-142-246 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:11:15.855353 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.855260    2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:11:15.862734 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862713    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:15.862734 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862733    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:15.862734 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862737    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:15.862734 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862740    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862743    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862746    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862749    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862751    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862754    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862756    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862759    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862762    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862764    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862767    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862769    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862772    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862775    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862778    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862780    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862783    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862785    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862788    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862790    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:15.862890 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862793    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862795    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862798    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862800    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862803    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862805    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862813    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862816    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862819    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862821    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862824    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862826    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862830    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862833    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862836    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862839    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862841    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862844    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862846    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862849    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:15.863383 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862852    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862854    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862856    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862859    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862861    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862864    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862866    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862869    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862871    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862874    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862876    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862879    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862881    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862884    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862886    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862891    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862895    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862900    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862904    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:15.863876 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862907    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862910    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862913    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862916    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862919    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862922    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862925    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862928    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862931    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862934    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862936    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862939    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862941    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862944    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862946    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862949    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862951    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862954    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862956    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862959    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:15.864346 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862961    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862965    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862968    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862970    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862972    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862976    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862980    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862985    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862988    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862991    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862994    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862996    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.862999    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863002    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863005    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863007    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863010    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863722    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863731    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:15.864834 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863735    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863738    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863741    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863743    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863746    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863748    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863751    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863756    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863759    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863762    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863765    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863768    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863771    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863773    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863776    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863779    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863781    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863784    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863787    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:15.865296 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863789    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863792    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863794    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863796    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863799    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863802    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863804    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863807    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863810    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863813    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863815    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863818    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863821    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863824    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863826    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863829    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863831    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863834    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863837    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863839    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:15.865784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863842    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863844    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863847    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863850    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863852    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863855    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863857    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863859    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863862    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863865    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863868    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863870    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863873    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863875    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863877    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863880    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863883    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863885    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863888    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:15.866278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863891    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863893    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863896    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863900    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863903    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863909    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863912    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863915    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863918    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863921    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863924    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863927    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863929    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863932    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863935    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863937    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863940    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863942    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863945    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:15.866766 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863947    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863950    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863952    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863954    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863957    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863960    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863962    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863965    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863967    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863970    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863972    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863975    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863977    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863980    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863982    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863985    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863988    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863991    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863995    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.863998    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:15.867337 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864388    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864393    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864396    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864399    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864403    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864406    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864408    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864412    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864415    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864419    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864423    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864427    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864430    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864435    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864439    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864441    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864445    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864448    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864450    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864453    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864455    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:15.867853 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864458    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864460    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864463    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864465    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864468    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864471    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864473    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864477    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864480    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864483    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864486    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864490    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864493    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864496    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864498    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864501    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864503    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864506    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864508    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:15.868534 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864511    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864514    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864516    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864519    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864521    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864524    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864526    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864529    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864531    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864537    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864539    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864542    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864544    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864547    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864549    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864552    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864554    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864557    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864559    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864561    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:15.868995 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864565    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864568    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864570    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864573    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864575    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864578    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864581    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864583    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864585    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864588    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864590    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864593    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864596    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864598    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864600    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864603    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864605    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864608    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864610    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864612    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:15.869518 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864615    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864619    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864622    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864625    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864628    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864631    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864635    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864637    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864640    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864642    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864645    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864648    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864651    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864654    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864656    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864659    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864661    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864664    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.864667    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864736    2571 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:11:15.869999 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864774    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864780    2571 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864784    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864788    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864791    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864795    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864799    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864802    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864804    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864808    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864811    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864814    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864817    2571 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864819    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864822    2571 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864826    2571 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864829    2571 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864833    2571 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864835    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864839    2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864842    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864845    2571 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864848    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:11:15.870519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864851    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864855    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864858    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864861    2571 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864864    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864866    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864869    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864873    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864877    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864880    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864882    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864885    2571 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864888    2571 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864890    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864895    2571 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864897    2571 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864900    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864903    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864906    2571 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864909    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864912    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864914    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864917    2571 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864920    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864923    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:11:15.871087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864932    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864935    2571 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864937    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864940    2571 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864943    2571 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864951    2571 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864953    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864956    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864960    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864963    2571 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864965    2571 flags.go:64] FLAG: --help="false"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864968    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-142-246.ec2.internal"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864971    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864974    2571 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864979    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864982    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864985    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864988    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864991    2571 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864993    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864996    2571 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.864999    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865002    2571 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865004    2571 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:11:15.871735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865007    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865010    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865012    2571 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865015    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865018    2571 flags.go:64] FLAG: --lock-file=""
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865021    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865023    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865026    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865032    2571 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865035    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865037    2571 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865040    2571 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865043    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865046    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865049    2571 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865051    2571 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865055    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865058    2571 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865062    2571 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865064    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865067    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865070    2571 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865074    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865077    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865080    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:11:15.872338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865082    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865089    2571 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865092    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865095    2571 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865098    2571 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865100    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865105    2571 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865108    2571 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865111    2571 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865113    2571 flags.go:64] FLAG: --port="10250"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865116    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865119    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0521e8c5dea92f6f4"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865122    2571 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865125    2571 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865127    2571 flags.go:64] FLAG: --register-node="true"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865130    2571 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865133    2571 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865136    2571 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865139    2571 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865142    2571 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865145    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865148    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865151    2571 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865154    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865156    2571 flags.go:64] FLAG: --runonce="false"
Sep 17 23:11:15.872951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865159    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865162    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865165    2571 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865167    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865171    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865174    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865177    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865180    2571 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865182    2571 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865185    2571 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865188    2571 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865190    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865193    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865196    2571 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865198    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865203    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865206    2571 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865209    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865212    2571 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865215    2571 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865217    2571 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865220    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865223    2571 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865226    2571 flags.go:64] FLAG: --v="2"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865229    2571 flags.go:64] FLAG: --version="false"
Sep 17 23:11:15.873562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865234    2571 flags.go:64] FLAG: --vmodule=""
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865237    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.865240    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865367    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865373    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865376    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865379    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865382    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865384    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865387    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865389    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865393    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865396    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865398    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865402    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865406    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865409    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865411    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865415    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865417    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:15.874203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865420    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865423    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865425    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865428    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865430    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865433    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865435    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865438    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865440    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865442    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865445    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865448    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865450    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865453    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865456    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865458    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865462    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865464    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865467    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:15.874718 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865469    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865472    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865474    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865478    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865483    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865486    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865488    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865491    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865493    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865496    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865498    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865501    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865503    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865506    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865508    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865511    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865513    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865516    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865518    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865521    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:15.875187 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865523    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865526    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865528    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865531    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865533    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865535    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865539    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865543    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865546    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865551    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865554    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865557    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865560    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865562    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865565    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865567    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865572    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865574    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865577    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865579    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:15.875683 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865582    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865584    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865587    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865589    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865592    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865595    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865597    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865600    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865602    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865604    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865607    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865609    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865612    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865614    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865617    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865619    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865622    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865624    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865627    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:15.876170 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865629    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865632    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865636    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.865639    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.866477    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.874355    2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.874370    2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.874383    2571 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.874389    2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874445    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874449    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874452    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874455    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874458    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874461    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:15.876651 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874463    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874466    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874468    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874472    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874476    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874479    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874483    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874487    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874491    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874495    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874498    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874502    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874505    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874507    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874510    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874512    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874515    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874518    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874521    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874523    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:15.877030 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874526    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874529    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874531    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874535    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874537    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874540    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874544    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874547    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874550    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874553    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874556    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874558    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874561    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874564    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874567    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874569    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874572    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874575    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874579    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:15.877562 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874582    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874585    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874587    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874589    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874592    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874602    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874606    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874609    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874611    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874614    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874617    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874619    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874622    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874625    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874627    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874630    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874633    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874637    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874639    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874642    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:15.878022 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874644    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874647    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874649    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874652    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874654    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874656    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874659    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874661    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874664    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874666    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874668    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874671    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874673    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874676    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874678    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874681    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874683    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874686    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874688    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:15.878515 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874691    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874693    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874696    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874698    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874700    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874703    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874705    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874708    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874710    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874713    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874715    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874718    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874721    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874724    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874726    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:15.878984 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.874732    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874840    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874844    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874847    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874850    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874852    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874855    2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874858    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874860    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874863    2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874866    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874868    2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874871    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874873    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874876    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874879    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874881    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874884    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874887    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:11:15.879415 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874890    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874892    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874895    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874897    2571 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874900    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874902    2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874905    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874907    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874911    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874914    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874917    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874919    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874922    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874926    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874929    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874931    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874934    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874936    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874939    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874941    2571 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:11:15.879870 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874944    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874947    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874949    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874952    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874954    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874957    2571 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874959    2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874962    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874964    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874966    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874969    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874972    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874974    2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874977    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874979    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874981    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874984    2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874987    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874990    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874992    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:11:15.880357 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874995    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.874997    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875000    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875003    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875005    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875009    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875012    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875014    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875017    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875020    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875022    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875025    2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875027    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875029    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875032    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875034    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875037    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875039    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875042    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875044    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:11:15.880833 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875047    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875049    2571 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875052    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875054    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875057    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875059    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875061    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875064    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875066    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875070    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875074    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875078    2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875081    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875084    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875087    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875090    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875093    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875096    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875099    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:11:15.881288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875102    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:11:15.881767 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:15.875105    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:11:15.881767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.875110    2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:11:15.881767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.875202    2571 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:11:15.881767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.877546    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:11:15.881767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.878581    2571 server.go:1019] "Starting client certificate rotation"
Sep 17 23:11:15.881767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.878688    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:11:15.881767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.878723    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:11:15.881767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.881598    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:11:15.886875 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.886858    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:11:15.898586 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.898569    2571 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:11:15.900373 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.900360    2571 log.go:25] "Validated CRI v1 image API"
Sep 17 23:11:15.901139 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.901125    2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:11:15.904181 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.904162    2571 fs.go:135] Filesystem UUIDs: map[35fe4ccb-35f6-4ce0-ac6b-b6c3caa69529:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 94e9d67b-e6ce-40b4-ae91-62d8f4babf78:/dev/nvme0n1p3]
Sep 17 23:11:15.904251 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.904181    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:11:15.907009 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.906990    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:11:15.909689 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.909578    2571 manager.go:216] Machine: {Timestamp:2026-09-17 23:11:15.908089615 +0000 UTC m=+0.369007337 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3053568 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec216a3c87cefd121f579843f58c8888 SystemUUID:ec216a3c-87ce-fd12-1f57-9843f58c8888 BootID:90ee8420-842c-477b-bb97-4ee1ce341cc2 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:9e:b2:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:9e:b2:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:09:14:3d:bd:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:11:15.909689 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.909682    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:11:15.909843 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.909777    2571 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:11:15.913448 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.913425    2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:11:15.913573 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.913451    2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-246.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:11:15.914305 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.914295    2571 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:11:15.914366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.914308    2571 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:11:15.914366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.914315    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:11:15.914366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.914343    2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:11:15.914451 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.914408    2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:11:15.920303 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.920292    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:11:15.920426 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.920417    2571 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:11:15.920460 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.920447    2571 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:11:15.920496 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.920461    2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:11:15.920496 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.920478    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:11:15.920496 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.920490    2571 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:11:15.920595 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.920502    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:11:15.921063 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.921049    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:11:15.921104 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.921076    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:11:15.924043 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.924028    2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:11:15.926930 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.926916    2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:11:15.926980 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.926947    2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:11:15.928449 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928434    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928460    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928471    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928480    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928493    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928503    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928523    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928531    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928542    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928551    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928564    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:11:15.928566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.928572    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:11:15.930276 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.930263    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:11:15.930276 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.930278    2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:11:15.930708 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:15.930687    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:11:15.930768 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:15.930714    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:11:15.933992 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.933976    2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-246.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:11:15.934681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.934670    2571 server.go:1268] "Started kubelet"
Sep 17 23:11:15.934796 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.934758    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:11:15.934858 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.934764    2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:11:15.934858 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.934829    2571 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:11:15.935566 ip-10-0-142-246 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:11:15.936257 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.936166    2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:11:15.936954 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.936904    2571 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:11:15.939346 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.938846    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-242mq"
Sep 17 23:11:15.941226 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:15.941196    2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:11:15.944466 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:15.943312    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-246.ec2.internal.18d63e8cb255610a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-246.ec2.internal,UID:ip-10-0-142-246.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-246.ec2.internal,},FirstTimestamp:2026-09-17 23:11:15.93463425 +0000 UTC m=+0.395551971,LastTimestamp:2026-09-17 23:11:15.93463425 +0000 UTC m=+0.395551971,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-246.ec2.internal,}"
Sep 17 23:11:15.946198 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.946183    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:11:15.946881 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.946865    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:11:15.947035 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:15.947010    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:15.947802 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.947744    2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:11:15.947889 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.947777    2571 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:11:15.947942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.947895    2571 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:11:15.947987 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.947956    2571 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:11:15.947987 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.947967    2571 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:11:15.948195 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.948180    2571 factory.go:153] Registering CRI-O factory
Sep 17 23:11:15.948259 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.948199    2571 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:11:15.948259 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.948253    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:11:15.948368 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.948262    2571 factory.go:55] Registering systemd factory
Sep 17 23:11:15.948368 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.948270    2571 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:11:15.948368 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.948288    2571 factory.go:103] Registering Raw factory
Sep 17 23:11:15.948368 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.948302    2571 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:11:15.948736 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.948723    2571 manager.go:349] Starting recovery of all containers
Sep 17 23:11:15.949961 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.949720    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-242mq"
Sep 17 23:11:15.950673 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:15.950651    2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:11:15.950764 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:15.950680    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-246.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:11:15.960610 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.960592    2571 manager.go:354] Recovery completed
Sep 17 23:11:15.965353 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.965339    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:15.967642 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.967628    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:15.967694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.967658    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:15.967694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.967671    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:15.968193 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.968181    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:11:15.968193 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.968187    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:15.968193 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.968193    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:11:15.968300 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.968210    2571 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:11:15.969069 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:15.968970    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-246.ec2.internal.18d63e8cb44d08ba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-246.ec2.internal,UID:ip-10-0-142-246.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-246.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-246.ec2.internal,},FirstTimestamp:2026-09-17 23:11:15.967641786 +0000 UTC m=+0.428559508,LastTimestamp:2026-09-17 23:11:15.967641786 +0000 UTC m=+0.428559508,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-246.ec2.internal,}"
Sep 17 23:11:15.969293 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.969278    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:15.969385 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.969299    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:15.969385 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.969308    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:15.970186 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.970174    2571 policy_none.go:49] "None policy: Start"
Sep 17 23:11:15.970216 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.970191    2571 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:11:15.970216 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.970201    2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:11:15.972227 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:15.972216    2571 policy_none.go:47] "Start"
Sep 17 23:11:16.012977 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.012963    2571 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.013018    2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.013029    2571 server.go:87] "Starting device plugin registration server"
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.013261    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.013273    2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.013399    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.013480    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.013488    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.014971    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:11:16.029477 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.015014    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.051299 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.051268    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:11:16.052503 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.052486    2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:11:16.052569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.052526    2571 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:11:16.052569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.052546    2571 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:11:16.052647 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.052579    2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 23:11:16.054885 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.054864    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:16.114138 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.114096    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:16.114884 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.114870    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:16.114941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.114896    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:16.114941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.114906    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:16.114941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.114930    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.123779 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.123762    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.123820 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.123782    2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-246.ec2.internal\": node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.139333 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.139307    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.153316 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.153292    2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"]
Sep 17 23:11:16.153381 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.153363    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:16.154101 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.154086    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:16.154157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.154116    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:16.154157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.154126    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:16.156274 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.156262    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:16.156414 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.156398    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.156466 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.156434    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:16.157015 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.156999    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:16.157086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.157006    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:16.157086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.157030    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:16.157086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.157043    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:16.157086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.157047    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:16.157086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.157059    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:16.159199 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.159179    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.159288 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.159205    2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:11:16.159858 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.159843    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:11:16.159938 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.159864    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:11:16.159938 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.159878    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:11:16.188863 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.188848    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-246.ec2.internal\" not found" node="ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.193135 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.193120    2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-246.ec2.internal\" not found" node="ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.239902 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.239883    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.249552 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.249527    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0758ebe72a943e427e2e61837dacf5fc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-246.ec2.internal\" (UID: \"0758ebe72a943e427e2e61837dacf5fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.249601 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.249557    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5aac5fe8556640956c74d2813f2e142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal\" (UID: \"f5aac5fe8556640956c74d2813f2e142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.249601 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.249577    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5aac5fe8556640956c74d2813f2e142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal\" (UID: \"f5aac5fe8556640956c74d2813f2e142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.340234 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.340212    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.350583 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.350537    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5aac5fe8556640956c74d2813f2e142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal\" (UID: \"f5aac5fe8556640956c74d2813f2e142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.350641 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.350608    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0758ebe72a943e427e2e61837dacf5fc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-246.ec2.internal\" (UID: \"0758ebe72a943e427e2e61837dacf5fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.350641 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.350625    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5aac5fe8556640956c74d2813f2e142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal\" (UID: \"f5aac5fe8556640956c74d2813f2e142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.350718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.350635    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5aac5fe8556640956c74d2813f2e142-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal\" (UID: \"f5aac5fe8556640956c74d2813f2e142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.350718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.350670    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5aac5fe8556640956c74d2813f2e142-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal\" (UID: \"f5aac5fe8556640956c74d2813f2e142\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.350780 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.350744    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0758ebe72a943e427e2e61837dacf5fc-config\") pod \"kube-apiserver-proxy-ip-10-0-142-246.ec2.internal\" (UID: \"0758ebe72a943e427e2e61837dacf5fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.440983 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.440935    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.492452 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.492418    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.497945 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.497525    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:16.541937 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.541804    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.642295 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.642273    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.742783 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.742734    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.843256 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.843232    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.878769 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.878742    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:11:16.879162 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.878888    2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:11:16.944356 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:16.944313    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:16.959339 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.959295    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:06:16 +0000 UTC" deadline="2028-04-30 16:14:42.365043956 +0000 UTC"
Sep 17 23:11:16.959339 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:16.959340    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14177h3m25.405707849s"
Sep 17 23:11:17.045205 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:17.045159    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:17.094607 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:17.094506    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5aac5fe8556640956c74d2813f2e142.slice/crio-4a1181e60ec6f2c4fbade9fb0f3cdc5bf455a9ef6ae24bd0e16deb04e78c3f60 WatchSource:0}: Error finding container 4a1181e60ec6f2c4fbade9fb0f3cdc5bf455a9ef6ae24bd0e16deb04e78c3f60: Status 404 returned error can't find the container with id 4a1181e60ec6f2c4fbade9fb0f3cdc5bf455a9ef6ae24bd0e16deb04e78c3f60
Sep 17 23:11:17.095266 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:17.095236    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0758ebe72a943e427e2e61837dacf5fc.slice/crio-fed89e606aa861b59cd18f501be1bc14337d63cab3ac1f37507a64706a972a2e WatchSource:0}: Error finding container fed89e606aa861b59cd18f501be1bc14337d63cab3ac1f37507a64706a972a2e: Status 404 returned error can't find the container with id fed89e606aa861b59cd18f501be1bc14337d63cab3ac1f37507a64706a972a2e
Sep 17 23:11:17.100449 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.100435    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:11:17.146152 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:17.146130    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:17.246610 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:17.246581    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:17.346839 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:17.346782    2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-246.ec2.internal\" not found"
Sep 17 23:11:17.351080 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.351062    2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:17.392297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.392274    2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:17.435608 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.435586    2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:17.447755 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.447733    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:17.458009 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.457990    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:11:17.458851 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.458838    2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal"
Sep 17 23:11:17.469377 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.469359    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:11:17.912041 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.912010    2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:11:17.921447 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.921424    2571 apiserver.go:52] "Watching apiserver"
Sep 17 23:11:17.928399 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.928375    2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:11:17.928861 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.928835    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-jsdb7","kube-system/konnectivity-agent-629tc","openshift-cluster-node-tuning-operator/tuned-47bgg","openshift-dns/node-resolver-ztfrq","openshift-image-registry/node-ca-scwvj","openshift-multus/multus-5w888","openshift-multus/network-metrics-daemon-jfgnd","openshift-network-diagnostics/network-check-target-b6w4t","openshift-network-operator/iptables-alerter-rfhbd","kube-system/global-pull-secret-syncer-46w55","kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal","openshift-multus/multus-additional-cni-plugins-xjllt"]
Sep 17 23:11:17.931751 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.931724    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:17.931853 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:17.931820    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:17.934087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.934062    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:17.934199 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:17.934156    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:17.936489 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.936462    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.938905 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.938884    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:17.939007 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.938971    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dfx24\""
Sep 17 23:11:17.939364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.939318    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:11:17.939364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.939344    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:11:17.941500 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.941480    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:17.941845 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.941822    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:11:17.941934 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.941877    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:11:17.942046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.942024    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v6jtc\""
Sep 17 23:11:17.943839 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.943763    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.944596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.944376    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:11:17.944596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.944398    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:11:17.944596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.944470    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vspd5\""
Sep 17 23:11:17.944596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.944358    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:11:17.946118 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.946099    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:17.946420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.946371    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:11:17.946804 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.946785    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:11:17.946898 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.946888    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:11:17.946975 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.946914    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:11:17.947081 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.947057    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b42bs\""
Sep 17 23:11:17.947158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.947091    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:11:17.948631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.948608    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:11:17.948820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.948802    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.950313 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.950288    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ln6jh\""
Sep 17 23:11:17.950461 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.950436    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:11:17.950538 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.950518    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:11:17.951338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.951317    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:17.954468 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.954033    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:11:17.954468 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.954254    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:11:17.954653 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.954633    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nqsjd\""
Sep 17 23:11:17.954757 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.954740    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:11:17.954894 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.954881    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:11:17.954955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.954903    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:11:17.955139 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.955121    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:11:17.955579 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.955554    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:11:17.956038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.955788    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:17.956038 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:17.955920    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:17.956187 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.956049    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hxqrc\""
Sep 17 23:11:17.956187 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.955793    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:11:17.958264 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.958247    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babd962d-1045-4ab4-8466-e347eeb95299-tmp\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.958387 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.958275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-slash\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.958387 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.958306    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-env-overrides\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.958675 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.958640    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:17.960015 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.959237    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovnkube-script-lib\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.960015 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.959946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-cni-bin\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.960015 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.959972    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-systemd\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.960426 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960374    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-cni-bin\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960583    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-cnibin\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960647    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-465hm\" (UniqueName: \"kubernetes.io/projected/14b269cf-41fe-45d2-b1aa-1957dd795540-kube-api-access-465hm\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960671    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2600b302-c671-4f21-9817-6bf74a225573-host-slash\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960694    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fdls\" (UniqueName: \"kubernetes.io/projected/243eb096-f63f-4cb7-8e3a-e82e582b400b-kube-api-access-4fdls\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960718    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-etc-kubernetes\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960740    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960761    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-daemon-config\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ch8\" (UniqueName: \"kubernetes.io/projected/babd962d-1045-4ab4-8466-e347eeb95299-kube-api-access-c7ch8\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-var-lib-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-hostroot\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960889    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960913    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960939    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysctl-conf\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960962    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovnkube-config\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.960987    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrcht\" (UniqueName: \"kubernetes.io/projected/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-kube-api-access-hrcht\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522d8a18-aacb-42d7-847e-add2e6355433-tmp-dir\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmbh\" (UniqueName: \"kubernetes.io/projected/522d8a18-aacb-42d7-847e-add2e6355433-kube-api-access-jqmbh\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-kubelet\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961101    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-node-log\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961124    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-log-socket\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961146    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-cni-netd\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961171    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-systemd\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-run-netns\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961220    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961277    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-socket-dir-parent\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961676    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysconfig\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961699    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/522d8a18-aacb-42d7-847e-add2e6355433-hosts-file\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961727    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961738    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ggcd\" (UniqueName: \"kubernetes.io/projected/e0cba546-5344-448a-bed8-1252738e58e2-kube-api-access-8ggcd\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961770    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-ovn\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961784    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovn-node-metrics-cert\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961797    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-netns\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.962998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961810    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-conf-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961825    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-k8s-cni-cncf-io\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-multus-certs\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961857    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-modprobe-d\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961872    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-cni-multus\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961875    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7wc8s\""
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961924    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5sc\" (UniqueName: \"kubernetes.io/projected/2600b302-c671-4f21-9817-6bf74a225573-kube-api-access-sq5sc\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961961    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-kubernetes\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.961991    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-lib-modules\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962028    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-etc-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962053    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962076    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-cni-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962119    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-kubelet\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962144    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2600b302-c671-4f21-9817-6bf74a225573-iptables-alerter-script\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962170    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0cba546-5344-448a-bed8-1252738e58e2-host\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962194    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-systemd-units\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962216    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-system-cni-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.963973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962239    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysctl-d\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962284    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-run\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962307    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-sys\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/babd962d-1045-4ab4-8466-e347eeb95299-etc-tuned\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962354    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0cba546-5344-448a-bed8-1252738e58e2-serviceca\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962395    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b269cf-41fe-45d2-b1aa-1957dd795540-cni-binary-copy\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-var-lib-kubelet\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962490    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.962501    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-host\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.963949    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:17.964911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.963996    2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:11:17.966676 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.966655    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:11:17.966994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.966949    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:11:17.966994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.967008    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hmcg4\""
Sep 17 23:11:17.968182 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.968166    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:11:17.992108 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.992083    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9kj5x"
Sep 17 23:11:18.000004 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:17.999973    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9kj5x"
Sep 17 23:11:18.049661 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.049634    2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:11:18.058297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.058247    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal" event={"ID":"f5aac5fe8556640956c74d2813f2e142","Type":"ContainerStarted","Data":"4a1181e60ec6f2c4fbade9fb0f3cdc5bf455a9ef6ae24bd0e16deb04e78c3f60"}
Sep 17 23:11:18.059498 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.059473    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal" event={"ID":"0758ebe72a943e427e2e61837dacf5fc","Type":"ContainerStarted","Data":"fed89e606aa861b59cd18f501be1bc14337d63cab3ac1f37507a64706a972a2e"}
Sep 17 23:11:18.062807 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.062785    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovnkube-config\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.062914 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.062820    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrcht\" (UniqueName: \"kubernetes.io/projected/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-kube-api-access-hrcht\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.062914 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.062841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.062914 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.062859    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-sys-fs\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.062914 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.062896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdpzq\" (UniqueName: \"kubernetes.io/projected/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-kube-api-access-fdpzq\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.063128 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.062924    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522d8a18-aacb-42d7-847e-add2e6355433-tmp-dir\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:18.063128 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.062948    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmbh\" (UniqueName: \"kubernetes.io/projected/522d8a18-aacb-42d7-847e-add2e6355433-kube-api-access-jqmbh\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:18.063128 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.062978    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-kubelet\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063128 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-node-log\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063128 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063030    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-log-socket\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063128 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063055    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-cni-netd\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063167    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-systemd\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063195    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-run-netns\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063221    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063248    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-socket-dir-parent\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysconfig\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063325    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/522d8a18-aacb-42d7-847e-add2e6355433-hosts-file\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063363    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ggcd\" (UniqueName: \"kubernetes.io/projected/e0cba546-5344-448a-bed8-1252738e58e2-kube-api-access-8ggcd\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063333    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-log-socket\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-ovn\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063454    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovn-node-metrics-cert\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063520    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-netns\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063544    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-conf-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063581    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14398bb6-0891-4bde-a292-08c56c4d2341-konnectivity-ca\") pod \"konnectivity-agent-629tc\" (UID: \"14398bb6-0891-4bde-a292-08c56c4d2341\") " pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063644    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-k8s-cni-cncf-io\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063668    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-multus-certs\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063696    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-modprobe-d\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063720    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-cni-multus\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063734    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovnkube-config\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063790    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522d8a18-aacb-42d7-847e-add2e6355433-tmp-dir\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063841    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-registration-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063883    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-kubelet\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063893    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14398bb6-0891-4bde-a292-08c56c4d2341-agent-certs\") pod \"konnectivity-agent-629tc\" (UID: \"14398bb6-0891-4bde-a292-08c56c4d2341\") " pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:18.063903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-node-log\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063963    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.063993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-systemd\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064028    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5sc\" (UniqueName: \"kubernetes.io/projected/2600b302-c671-4f21-9817-6bf74a225573-kube-api-access-sq5sc\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064055    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-netns\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064087    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-kubernetes\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064118    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-lib-modules\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064117    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-cni-netd\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064149    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-etc-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064184    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-conf-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064220    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-cni-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064279    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-kubelet\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064284    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-k8s-cni-cncf-io\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064312    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064333    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-run-multus-certs\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064342    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2600b302-c671-4f21-9817-6bf74a225573-iptables-alerter-script\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:18.064630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064387    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0cba546-5344-448a-bed8-1252738e58e2-host\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064439    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-systemd-units\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064483    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-system-cni-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064533    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26zxr\" (UniqueName: \"kubernetes.io/projected/bb310836-ff74-44fd-9ed2-2c624875cc9b-kube-api-access-26zxr\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064516    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-modprobe-d\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064563    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysctl-d\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064562    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-run-netns\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064586    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-cni-multus\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064609    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064653    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-etc-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064658    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-kubernetes\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064697    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064721    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-lib-modules\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064861    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-socket-dir-parent\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064890    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-cni-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064919    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysconfig\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064955    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-kubelet\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064970    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/522d8a18-aacb-42d7-847e-add2e6355433-hosts-file\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:18.065535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065075    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-ovn\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065303    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-run\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065330    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2600b302-c671-4f21-9817-6bf74a225573-iptables-alerter-script\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065342    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-sys\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.064485    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065429    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-sys\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065527    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-run\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065580    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/babd962d-1045-4ab4-8466-e347eeb95299-etc-tuned\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065625    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0cba546-5344-448a-bed8-1252738e58e2-serviceca\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065640    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysctl-d\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b269cf-41fe-45d2-b1aa-1957dd795540-cni-binary-copy\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065696    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065717    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-system-cni-dir\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065738    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-system-cni-dir\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065769    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-var-lib-kubelet\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065785    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-host\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065777    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-systemd-units\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065837    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babd962d-1045-4ab4-8466-e347eeb95299-tmp\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.066380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065868    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-slash\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065909    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-env-overrides\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065976    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovnkube-script-lib\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066028    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-cni-bin\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066060    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066107    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-systemd\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-cni-bin\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066181    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-cnibin\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066204    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-465hm\" (UniqueName: \"kubernetes.io/projected/14b269cf-41fe-45d2-b1aa-1957dd795540-kube-api-access-465hm\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066278    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-host\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066288    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-socket-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-slash\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066374    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e0cba546-5344-448a-bed8-1252738e58e2-serviceca\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066332    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85f4d3fc-3215-4e00-b5e1-175750c70026-kubelet-config\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.065907    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-var-lib-kubelet\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066456    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e0cba546-5344-448a-bed8-1252738e58e2-host\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:18.067431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/14b269cf-41fe-45d2-b1aa-1957dd795540-cni-binary-copy\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066494    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2600b302-c671-4f21-9817-6bf74a225573-host-slash\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066565    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fdls\" (UniqueName: \"kubernetes.io/projected/243eb096-f63f-4cb7-8e3a-e82e582b400b-kube-api-access-4fdls\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066593    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-cni-bin\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066620    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-etc-kubernetes\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066639    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-host-var-lib-cni-bin\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066648    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066676    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-run-systemd\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066677    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-daemon-config\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85f4d3fc-3215-4e00-b5e1-175750c70026-dbus\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066737    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-cnibin\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066758    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-cnibin\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066802    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066801    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-etc-kubernetes\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066841    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ch8\" (UniqueName: \"kubernetes.io/projected/babd962d-1045-4ab4-8466-e347eeb95299-kube-api-access-c7ch8\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066866    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-var-lib-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066854    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2600b302-c671-4f21-9817-6bf74a225573-host-slash\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:18.068292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066883    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-hostroot\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.069123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066918    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-device-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.069123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066946    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:18.069123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066958    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-var-lib-openvswitch\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.069123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066972    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:18.069123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.067007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysctl-conf\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.069123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.068520    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovnkube-script-lib\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.069123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.068755    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/14b269cf-41fe-45d2-b1aa-1957dd795540-multus-daemon-config\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.069551 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.069300    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-host-run-ovn-kubernetes\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.069551 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.069377    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/babd962d-1045-4ab4-8466-e347eeb95299-etc-sysctl-conf\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.069551 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.066880    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-env-overrides\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.069832 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.069815    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:18.070004 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.069993    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs podName:243eb096-f63f-4cb7-8e3a-e82e582b400b nodeName:}" failed. No retries permitted until 2026-09-17 23:11:18.569971567 +0000 UTC m=+3.030889304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs") pod "network-metrics-daemon-jfgnd" (UID: "243eb096-f63f-4cb7-8e3a-e82e582b400b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:18.070349 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.070301    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/14b269cf-41fe-45d2-b1aa-1957dd795540-hostroot\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.072684 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.070598    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/babd962d-1045-4ab4-8466-e347eeb95299-etc-tuned\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.072684 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.071013    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-ovn-node-metrics-cert\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.074303 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.074282    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/babd962d-1045-4ab4-8466-e347eeb95299-tmp\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.078102 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.078078    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmbh\" (UniqueName: \"kubernetes.io/projected/522d8a18-aacb-42d7-847e-add2e6355433-kube-api-access-jqmbh\") pod \"node-resolver-ztfrq\" (UID: \"522d8a18-aacb-42d7-847e-add2e6355433\") " pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:18.079174 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.079152    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:18.079284 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.079178    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:18.079284 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.079193    2571 projected.go:196] Error preparing data for projected volume kube-api-access-frg99 for pod openshift-network-diagnostics/network-check-target-b6w4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:18.079284 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.079252    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99 podName:960a9634-3ae7-41f4-83be-a9b9f3d008a7 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:18.579234269 +0000 UTC m=+3.040151980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-frg99" (UniqueName: "kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99") pod "network-check-target-b6w4t" (UID: "960a9634-3ae7-41f4-83be-a9b9f3d008a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:18.080066 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.080005    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrcht\" (UniqueName: \"kubernetes.io/projected/4d6e87a1-b303-4d53-ac3e-49acdca61c0d-kube-api-access-hrcht\") pod \"ovnkube-node-jsdb7\" (UID: \"4d6e87a1-b303-4d53-ac3e-49acdca61c0d\") " pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.080920 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.080895    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ch8\" (UniqueName: \"kubernetes.io/projected/babd962d-1045-4ab4-8466-e347eeb95299-kube-api-access-c7ch8\") pod \"tuned-47bgg\" (UID: \"babd962d-1045-4ab4-8466-e347eeb95299\") " pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.082185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.082166    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ggcd\" (UniqueName: \"kubernetes.io/projected/e0cba546-5344-448a-bed8-1252738e58e2-kube-api-access-8ggcd\") pod \"node-ca-scwvj\" (UID: \"e0cba546-5344-448a-bed8-1252738e58e2\") " pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:18.083216 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.082894    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fdls\" (UniqueName: \"kubernetes.io/projected/243eb096-f63f-4cb7-8e3a-e82e582b400b-kube-api-access-4fdls\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:18.083216 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.083179    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-465hm\" (UniqueName: \"kubernetes.io/projected/14b269cf-41fe-45d2-b1aa-1957dd795540-kube-api-access-465hm\") pod \"multus-5w888\" (UID: \"14b269cf-41fe-45d2-b1aa-1957dd795540\") " pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.083590 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.083572    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5sc\" (UniqueName: \"kubernetes.io/projected/2600b302-c671-4f21-9817-6bf74a225573-kube-api-access-sq5sc\") pod \"iptables-alerter-rfhbd\" (UID: \"2600b302-c671-4f21-9817-6bf74a225573\") " pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:18.169157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169078    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14398bb6-0891-4bde-a292-08c56c4d2341-konnectivity-ca\") pod \"konnectivity-agent-629tc\" (UID: \"14398bb6-0891-4bde-a292-08c56c4d2341\") " pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:18.169157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169125    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-registration-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.169157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169150    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14398bb6-0891-4bde-a292-08c56c4d2341-agent-certs\") pod \"konnectivity-agent-629tc\" (UID: \"14398bb6-0891-4bde-a292-08c56c4d2341\") " pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169176    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169212    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169228    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-registration-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169241    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26zxr\" (UniqueName: \"kubernetes.io/projected/bb310836-ff74-44fd-9ed2-2c624875cc9b-kube-api-access-26zxr\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169294    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169358    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-system-cni-dir\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169391    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169421    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-socket-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.169452    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169470    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85f4d3fc-3215-4e00-b5e1-175750c70026-kubelet-config\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.169533    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret podName:85f4d3fc-3215-4e00-b5e1-175750c70026 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:18.669512478 +0000 UTC m=+3.130430191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret") pod "global-pull-secret-syncer-46w55" (UID: "85f4d3fc-3215-4e00-b5e1-175750c70026") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169556    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85f4d3fc-3215-4e00-b5e1-175750c70026-dbus\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169561    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-cnibin\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.169697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169615    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-device-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169674    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-socket-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169695    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169721    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-sys-fs\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169723    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-cnibin\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdpzq\" (UniqueName: \"kubernetes.io/projected/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-kube-api-access-fdpzq\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169834    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14398bb6-0891-4bde-a292-08c56c4d2341-konnectivity-ca\") pod \"konnectivity-agent-629tc\" (UID: \"14398bb6-0891-4bde-a292-08c56c4d2341\") " pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169901    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169923    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/85f4d3fc-3215-4e00-b5e1-175750c70026-dbus\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169931    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169969    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb310836-ff74-44fd-9ed2-2c624875cc9b-system-cni-dir\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169974    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-device-dir\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169990    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-sys-fs\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.169996    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.170002    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb310836-ff74-44fd-9ed2-2c624875cc9b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.170014    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.170431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.170050    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/85f4d3fc-3215-4e00-b5e1-175750c70026-kubelet-config\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.172013 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.171993    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14398bb6-0891-4bde-a292-08c56c4d2341-agent-certs\") pod \"konnectivity-agent-629tc\" (UID: \"14398bb6-0891-4bde-a292-08c56c4d2341\") " pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:18.179245 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.179220    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26zxr\" (UniqueName: \"kubernetes.io/projected/bb310836-ff74-44fd-9ed2-2c624875cc9b-kube-api-access-26zxr\") pod \"multus-additional-cni-plugins-xjllt\" (UID: \"bb310836-ff74-44fd-9ed2-2c624875cc9b\") " pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.179572 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.179553    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdpzq\" (UniqueName: \"kubernetes.io/projected/0acc5aa5-5b54-47fd-8c86-c9df335bcfb9-kube-api-access-fdpzq\") pod \"aws-ebs-csi-driver-node-dxvgl\" (UID: \"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.249192 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.249170    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-47bgg"
Sep 17 23:11:18.261816 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.261793    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ztfrq"
Sep 17 23:11:18.268712 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.268691    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-scwvj"
Sep 17 23:11:18.276433 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.276416    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5w888"
Sep 17 23:11:18.285209 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.285189    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rfhbd"
Sep 17 23:11:18.294480 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.294462    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:18.303472 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.303447    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:18.311385 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.311368    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xjllt"
Sep 17 23:11:18.317366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.317347    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl"
Sep 17 23:11:18.573927 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.573842    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:18.574088 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.573987    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:18.574088 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.574061    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs podName:243eb096-f63f-4cb7-8e3a-e82e582b400b nodeName:}" failed. No retries permitted until 2026-09-17 23:11:19.574042921 +0000 UTC m=+4.034960647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs") pod "network-metrics-daemon-jfgnd" (UID: "243eb096-f63f-4cb7-8e3a-e82e582b400b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:18.675221 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.675190    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:18.675409 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:18.675233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:18.675409 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.675383    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:18.675540 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.675408    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:18.675540 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.675432    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:18.675540 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.675446    2571 projected.go:196] Error preparing data for projected volume kube-api-access-frg99 for pod openshift-network-diagnostics/network-check-target-b6w4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:18.675540 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.675474    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret podName:85f4d3fc-3215-4e00-b5e1-175750c70026 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:19.67542956 +0000 UTC m=+4.136347286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret") pod "global-pull-secret-syncer-46w55" (UID: "85f4d3fc-3215-4e00-b5e1-175750c70026") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:18.675540 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:18.675500    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99 podName:960a9634-3ae7-41f4-83be-a9b9f3d008a7 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:19.675486426 +0000 UTC m=+4.136404137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-frg99" (UniqueName: "kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99") pod "network-check-target-b6w4t" (UID: "960a9634-3ae7-41f4-83be-a9b9f3d008a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:19.003522 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:19.003430    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:06:18 +0000 UTC" deadline="2028-03-14 11:08:56.546477695 +0000 UTC"
Sep 17 23:11:19.003522 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:19.003478    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13043h57m37.543011631s"
Sep 17 23:11:19.149819 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:19.149787    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acc5aa5_5b54_47fd_8c86_c9df335bcfb9.slice/crio-0c24695f79e639326896515f912d81efb5af61445b0fc88b3069e3ad364bdf03 WatchSource:0}: Error finding container 0c24695f79e639326896515f912d81efb5af61445b0fc88b3069e3ad364bdf03: Status 404 returned error can't find the container with id 0c24695f79e639326896515f912d81efb5af61445b0fc88b3069e3ad364bdf03
Sep 17 23:11:19.157264 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:19.157240    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b269cf_41fe_45d2_b1aa_1957dd795540.slice/crio-eeb4645dc6e515b8bc9b094c2845bd25b6355255a6314d7eb33af77c0934965a WatchSource:0}: Error finding container eeb4645dc6e515b8bc9b094c2845bd25b6355255a6314d7eb33af77c0934965a: Status 404 returned error can't find the container with id eeb4645dc6e515b8bc9b094c2845bd25b6355255a6314d7eb33af77c0934965a
Sep 17 23:11:19.158212 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:19.158155    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522d8a18_aacb_42d7_847e_add2e6355433.slice/crio-70900bd4dba5e0f3d2dcf2c89bdfa4eb22877f06b7d68ac20b33cd33ee3e0126 WatchSource:0}: Error finding container 70900bd4dba5e0f3d2dcf2c89bdfa4eb22877f06b7d68ac20b33cd33ee3e0126: Status 404 returned error can't find the container with id 70900bd4dba5e0f3d2dcf2c89bdfa4eb22877f06b7d68ac20b33cd33ee3e0126
Sep 17 23:11:19.159482 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:19.159283    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0cba546_5344_448a_bed8_1252738e58e2.slice/crio-338d957781c6775fe30a9ff6122d7c37623fd73cc9bf0ea076a59402d77b6449 WatchSource:0}: Error finding container 338d957781c6775fe30a9ff6122d7c37623fd73cc9bf0ea076a59402d77b6449: Status 404 returned error can't find the container with id 338d957781c6775fe30a9ff6122d7c37623fd73cc9bf0ea076a59402d77b6449
Sep 17 23:11:19.161278 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:19.161242    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6e87a1_b303_4d53_ac3e_49acdca61c0d.slice/crio-251028bf7dfeb31ba6ff085a8e2ea18a18da2c2cc29a2b908178e76fe71b70e9 WatchSource:0}: Error finding container 251028bf7dfeb31ba6ff085a8e2ea18a18da2c2cc29a2b908178e76fe71b70e9: Status 404 returned error can't find the container with id 251028bf7dfeb31ba6ff085a8e2ea18a18da2c2cc29a2b908178e76fe71b70e9
Sep 17 23:11:19.161923 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:19.161902    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14398bb6_0891_4bde_a292_08c56c4d2341.slice/crio-c659e32b388ea40c0ecb9414e7e1d7a6ec9f940b5abcb20bdd1e6476b2b0eeb0 WatchSource:0}: Error finding container c659e32b388ea40c0ecb9414e7e1d7a6ec9f940b5abcb20bdd1e6476b2b0eeb0: Status 404 returned error can't find the container with id c659e32b388ea40c0ecb9414e7e1d7a6ec9f940b5abcb20bdd1e6476b2b0eeb0
Sep 17 23:11:19.170436 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:19.170406    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb310836_ff74_44fd_9ed2_2c624875cc9b.slice/crio-54833a9d09a71bbde34927c7650f6ad1e936f9f1e0f2b8b6f9d5e548c6c1d772 WatchSource:0}: Error finding container 54833a9d09a71bbde34927c7650f6ad1e936f9f1e0f2b8b6f9d5e548c6c1d772: Status 404 returned error can't find the container with id 54833a9d09a71bbde34927c7650f6ad1e936f9f1e0f2b8b6f9d5e548c6c1d772
Sep 17 23:11:19.583768 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:19.583514    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:19.583929 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:19.583807    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:19.583929 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:19.583878    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs podName:243eb096-f63f-4cb7-8e3a-e82e582b400b nodeName:}" failed. No retries permitted until 2026-09-17 23:11:21.583855743 +0000 UTC m=+6.044773454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs") pod "network-metrics-daemon-jfgnd" (UID: "243eb096-f63f-4cb7-8e3a-e82e582b400b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:19.684575 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:19.684453    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:19.684575 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:19.684525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:19.684795 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:19.684696    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:19.684795 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:19.684716    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:19.684795 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:19.684730    2571 projected.go:196] Error preparing data for projected volume kube-api-access-frg99 for pod openshift-network-diagnostics/network-check-target-b6w4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:19.684795 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:19.684791    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99 podName:960a9634-3ae7-41f4-83be-a9b9f3d008a7 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:21.684771024 +0000 UTC m=+6.145688739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-frg99" (UniqueName: "kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99") pod "network-check-target-b6w4t" (UID: "960a9634-3ae7-41f4-83be-a9b9f3d008a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:19.685231 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:19.685209    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:19.685312 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:19.685270    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret podName:85f4d3fc-3215-4e00-b5e1-175750c70026 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:21.685254295 +0000 UTC m=+6.146172011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret") pod "global-pull-secret-syncer-46w55" (UID: "85f4d3fc-3215-4e00-b5e1-175750c70026") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:20.004603 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.004488    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:06:18 +0000 UTC" deadline="2028-04-10 05:39:15.156663775 +0000 UTC"
Sep 17 23:11:20.004603 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.004554    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13686h27m55.152114445s"
Sep 17 23:11:20.055551 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.055523    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:20.055724 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:20.055655    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:20.056096 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.056071    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:20.056224 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:20.056201    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:20.056776 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.056754    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:20.056871 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:20.056856    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:20.089144 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.088003    2571 generic.go:356] "Generic (PLEG): container finished" podID="f5aac5fe8556640956c74d2813f2e142" containerID="40477df1de50bf9901b1b391d47aff3bfaedd3cd227522e797f51bfb25b739fa" exitCode=0
Sep 17 23:11:20.089144 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.088966    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal" event={"ID":"f5aac5fe8556640956c74d2813f2e142","Type":"ContainerDied","Data":"40477df1de50bf9901b1b391d47aff3bfaedd3cd227522e797f51bfb25b739fa"}
Sep 17 23:11:20.111157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.111128    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal" event={"ID":"0758ebe72a943e427e2e61837dacf5fc","Type":"ContainerStarted","Data":"f75da1741a27d545eba1416444675a2caaf5b50e87ffa4c2f56f7283ed5f5bca"}
Sep 17 23:11:20.128937 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.128904    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rfhbd" event={"ID":"2600b302-c671-4f21-9817-6bf74a225573","Type":"ContainerStarted","Data":"33cafbde11626f8b0219f1882fd8bbf0380bdffa85fbabcc637d64355207885f"}
Sep 17 23:11:20.130061 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.130010    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-246.ec2.internal" podStartSLOduration=3.129990513 podStartE2EDuration="3.129990513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:20.129759335 +0000 UTC m=+4.590677070" watchObservedRunningTime="2026-09-17 23:11:20.129990513 +0000 UTC m=+4.590908243"
Sep 17 23:11:20.142788 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.142747    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-47bgg" event={"ID":"babd962d-1045-4ab4-8466-e347eeb95299","Type":"ContainerStarted","Data":"6f1b11c4008139f90bb349c7a34f30b38cd329d015a765eba8390f7380d23658"}
Sep 17 23:11:20.162366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.162322    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"251028bf7dfeb31ba6ff085a8e2ea18a18da2c2cc29a2b908178e76fe71b70e9"}
Sep 17 23:11:20.175916 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.175839    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-scwvj" event={"ID":"e0cba546-5344-448a-bed8-1252738e58e2","Type":"ContainerStarted","Data":"338d957781c6775fe30a9ff6122d7c37623fd73cc9bf0ea076a59402d77b6449"}
Sep 17 23:11:20.192136 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.192075    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl" event={"ID":"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9","Type":"ContainerStarted","Data":"0c24695f79e639326896515f912d81efb5af61445b0fc88b3069e3ad364bdf03"}
Sep 17 23:11:20.212297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.212269    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xjllt" event={"ID":"bb310836-ff74-44fd-9ed2-2c624875cc9b","Type":"ContainerStarted","Data":"54833a9d09a71bbde34927c7650f6ad1e936f9f1e0f2b8b6f9d5e548c6c1d772"}
Sep 17 23:11:20.226492 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.226466    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-629tc" event={"ID":"14398bb6-0891-4bde-a292-08c56c4d2341","Type":"ContainerStarted","Data":"c659e32b388ea40c0ecb9414e7e1d7a6ec9f940b5abcb20bdd1e6476b2b0eeb0"}
Sep 17 23:11:20.241295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.241268    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ztfrq" event={"ID":"522d8a18-aacb-42d7-847e-add2e6355433","Type":"ContainerStarted","Data":"70900bd4dba5e0f3d2dcf2c89bdfa4eb22877f06b7d68ac20b33cd33ee3e0126"}
Sep 17 23:11:20.245809 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:20.245784    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5w888" event={"ID":"14b269cf-41fe-45d2-b1aa-1957dd795540","Type":"ContainerStarted","Data":"eeb4645dc6e515b8bc9b094c2845bd25b6355255a6314d7eb33af77c0934965a"}
Sep 17 23:11:21.275587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:21.275484    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal" event={"ID":"f5aac5fe8556640956c74d2813f2e142","Type":"ContainerStarted","Data":"d0bde36ef90777d72a4ab377190466a111f3ba78916815420da1b1358984f637"}
Sep 17 23:11:21.603655 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:21.602975    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:21.603655 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:21.603140    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:21.603655 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:21.603206    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs podName:243eb096-f63f-4cb7-8e3a-e82e582b400b nodeName:}" failed. No retries permitted until 2026-09-17 23:11:25.603185022 +0000 UTC m=+10.064102733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs") pod "network-metrics-daemon-jfgnd" (UID: "243eb096-f63f-4cb7-8e3a-e82e582b400b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:21.704465 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:21.703702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:21.704465 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:21.703772    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:21.704465 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:21.703931    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:21.704465 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:21.703948    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:21.704465 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:21.703958    2571 projected.go:196] Error preparing data for projected volume kube-api-access-frg99 for pod openshift-network-diagnostics/network-check-target-b6w4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:21.704465 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:21.704011    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99 podName:960a9634-3ae7-41f4-83be-a9b9f3d008a7 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:25.703990946 +0000 UTC m=+10.164908675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-frg99" (UniqueName: "kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99") pod "network-check-target-b6w4t" (UID: "960a9634-3ae7-41f4-83be-a9b9f3d008a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:21.704465 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:21.704087    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:21.704465 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:21.704123    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret podName:85f4d3fc-3215-4e00-b5e1-175750c70026 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:25.704111714 +0000 UTC m=+10.165029433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret") pod "global-pull-secret-syncer-46w55" (UID: "85f4d3fc-3215-4e00-b5e1-175750c70026") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:22.056682 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:22.055974    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:22.056682 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:22.056100    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:22.056682 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:22.056500    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:22.056682 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:22.056592    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:22.056682 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:22.056632    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:22.057061 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:22.056758    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:24.055160 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:24.055117    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:24.055679 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:24.055253    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:24.055679 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:24.055330    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:24.055679 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:24.055387    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:24.055679 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:24.055486    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:24.055679 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:24.055249    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:25.634651 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:25.634619    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:25.635108 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:25.634749    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:25.635108 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:25.634805    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs podName:243eb096-f63f-4cb7-8e3a-e82e582b400b nodeName:}" failed. No retries permitted until 2026-09-17 23:11:33.634787005 +0000 UTC m=+18.095704728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs") pod "network-metrics-daemon-jfgnd" (UID: "243eb096-f63f-4cb7-8e3a-e82e582b400b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:25.735130 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:25.735099    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:25.735313 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:25.735159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:25.735313 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:25.735275    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:25.735313 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:25.735299    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:25.735313 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:25.735315    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:25.735550 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:25.735362    2571 projected.go:196] Error preparing data for projected volume kube-api-access-frg99 for pod openshift-network-diagnostics/network-check-target-b6w4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:25.735550 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:25.735366    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret podName:85f4d3fc-3215-4e00-b5e1-175750c70026 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:33.735346898 +0000 UTC m=+18.196264609 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret") pod "global-pull-secret-syncer-46w55" (UID: "85f4d3fc-3215-4e00-b5e1-175750c70026") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:25.735550 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:25.735412    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99 podName:960a9634-3ae7-41f4-83be-a9b9f3d008a7 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:33.735397154 +0000 UTC m=+18.196314877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-frg99" (UniqueName: "kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99") pod "network-check-target-b6w4t" (UID: "960a9634-3ae7-41f4-83be-a9b9f3d008a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:26.053450 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:26.053373    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:26.053595 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:26.053490    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:26.054592 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:26.054551    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:26.054714 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:26.054639    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:26.054714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:26.054695    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:26.054868 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:26.054776    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:28.054352 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:28.053664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:28.054352 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:28.053806    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:28.054352 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:28.054168    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:28.054352 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:28.054258    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:28.054897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:28.054392    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:28.054897 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:28.054506    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:30.053103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:30.053062    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:30.053515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:30.053078    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:30.053515 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:30.053190    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:30.053515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:30.053201    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:30.053515 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:30.053268    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:30.053515 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:30.053393    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:32.053747 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:32.053298    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:32.053747 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:32.053317    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:32.053747 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:32.053310    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:32.053747 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:32.053425    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:32.053747 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:32.053534    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:32.053747 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:32.053626    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:33.697138 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:33.697109    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:33.697572 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:33.697239    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:33.697572 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:33.697295    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs podName:243eb096-f63f-4cb7-8e3a-e82e582b400b nodeName:}" failed. No retries permitted until 2026-09-17 23:11:49.697274524 +0000 UTC m=+34.158192250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs") pod "network-metrics-daemon-jfgnd" (UID: "243eb096-f63f-4cb7-8e3a-e82e582b400b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:33.797988 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:33.797915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:33.797988 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:33.797979    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:33.798151 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:33.798063    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:33.798151 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:33.798085    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:33.798151 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:33.798106    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:33.798151 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:33.798118    2571 projected.go:196] Error preparing data for projected volume kube-api-access-frg99 for pod openshift-network-diagnostics/network-check-target-b6w4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:33.798151 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:33.798131    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret podName:85f4d3fc-3215-4e00-b5e1-175750c70026 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:49.798113654 +0000 UTC m=+34.259031382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret") pod "global-pull-secret-syncer-46w55" (UID: "85f4d3fc-3215-4e00-b5e1-175750c70026") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:33.798360 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:33.798164    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99 podName:960a9634-3ae7-41f4-83be-a9b9f3d008a7 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:49.798151585 +0000 UTC m=+34.259069296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-frg99" (UniqueName: "kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99") pod "network-check-target-b6w4t" (UID: "960a9634-3ae7-41f4-83be-a9b9f3d008a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:34.053003 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:34.052916    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:34.053164 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:34.052917    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:34.053164 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:34.053045    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:34.053164 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:34.053129    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:34.053164 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:34.052917    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:34.053405 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:34.053231    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:36.053694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:36.053664    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:36.054068 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:36.053750    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:36.054068 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:36.053838    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:36.054068 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:36.053914    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:36.054068 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:36.053955    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:36.054068 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:36.054001    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:38.053451 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.053230    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:38.054116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.053238    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:38.054116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.053292    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:38.054116 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:38.053623    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:38.054116 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:38.053587    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:38.054116 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:38.053671    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:38.308485 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.308444    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5w888" event={"ID":"14b269cf-41fe-45d2-b1aa-1957dd795540","Type":"ContainerStarted","Data":"30a89deca1520834195e948eef2b9e8c0aed6271ab531fa02a6c59ec0896974f"}
Sep 17 23:11:38.310659 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.310233    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-47bgg" event={"ID":"babd962d-1045-4ab4-8466-e347eeb95299","Type":"ContainerStarted","Data":"2b5c322cf284028835a95aad8df7dd7fe27416a19103d58a762e73ac347d7576"}
Sep 17 23:11:38.313878 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.313849    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"d15fb0f90db5a22d2141f445fad56cdf14000c2aea3a185d7ab896599b1d3611"}
Sep 17 23:11:38.314016 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.313935    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"2baa79ca3942bbe739ce96f7e49477e56eee3ca364a9bcf86a0415e478f8d05e"}
Sep 17 23:11:38.314016 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.313990    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"0ad0a11ce05833b7b511d31da286a9dcfb1467c613fc57372efd0c0de8c7d23f"}
Sep 17 23:11:38.314130 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.314033    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"0febc9de3695cb63f030347ad9bbb4c2311e2a2e33442b8bd43f0e3b2a62f6b1"}
Sep 17 23:11:38.314130 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.314080    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"ef094c8e90fe7d1020b42b6607eb10dea53d314b3b02a1dacccacf466b2df704"}
Sep 17 23:11:38.314233 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.314130    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"6139a84e2fbf228f27119ce2016266668590d3c277f32472d5e6093ecaa61629"}
Sep 17 23:11:38.315536 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.315510    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-scwvj" event={"ID":"e0cba546-5344-448a-bed8-1252738e58e2","Type":"ContainerStarted","Data":"e82902aac07780444d3b9919db81981121dfed044e09e91f6dcc03bc6620c936"}
Sep 17 23:11:38.317135 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.317109    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl" event={"ID":"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9","Type":"ContainerStarted","Data":"595f47c6e633ee86184922828d40771006d80164bb98df042deda6dd1b5ffee0"}
Sep 17 23:11:38.318784 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.318760    2571 generic.go:356] "Generic (PLEG): container finished" podID="bb310836-ff74-44fd-9ed2-2c624875cc9b" containerID="057852673a84e1c240976a7a9f9df9c492bc70fed6defa985ed0f0ef722d174d" exitCode=0
Sep 17 23:11:38.318912 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.318832    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xjllt" event={"ID":"bb310836-ff74-44fd-9ed2-2c624875cc9b","Type":"ContainerDied","Data":"057852673a84e1c240976a7a9f9df9c492bc70fed6defa985ed0f0ef722d174d"}
Sep 17 23:11:38.320472 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.320453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-629tc" event={"ID":"14398bb6-0891-4bde-a292-08c56c4d2341","Type":"ContainerStarted","Data":"2dce25945e5c28a3632fd1cf378fd41c1d8d8779e7a30eb61ce694f1f4c8d6d5"}
Sep 17 23:11:38.322094 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.322071    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ztfrq" event={"ID":"522d8a18-aacb-42d7-847e-add2e6355433","Type":"ContainerStarted","Data":"e31ab4c9cbbb39b1250793a0cfd1988ece8811de299d00ce4c2bb0fd1fff5d07"}
Sep 17 23:11:38.331028 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.330973    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5w888" podStartSLOduration=3.728074182 podStartE2EDuration="22.330956628s" totalImagesPullingTime="18.602882446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:38.330944348 +0000 UTC m=+22.791862074" watchObservedRunningTime="2026-09-17 23:11:38.330956628 +0000 UTC m=+22.791874360"
Sep 17 23:11:38.331375 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.331344    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-246.ec2.internal" podStartSLOduration=21.331314983 podStartE2EDuration="21.331314983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:21.302005414 +0000 UTC m=+5.762923150" watchObservedRunningTime="2026-09-17 23:11:38.331314983 +0000 UTC m=+22.792232713"
Sep 17 23:11:38.381731 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.381453    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ztfrq" podStartSLOduration=4.101615265 podStartE2EDuration="22.381435987s" totalImagesPullingTime="18.279820722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:38.38142569 +0000 UTC m=+22.842343420" watchObservedRunningTime="2026-09-17 23:11:38.381435987 +0000 UTC m=+22.842353715"
Sep 17 23:11:38.401711 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.401673    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-629tc" podStartSLOduration=4.197921436 podStartE2EDuration="22.401660274s" totalImagesPullingTime="18.203738838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:38.40160767 +0000 UTC m=+22.862525424" watchObservedRunningTime="2026-09-17 23:11:38.401660274 +0000 UTC m=+22.862578005"
Sep 17 23:11:38.425912 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:38.425875    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-47bgg" podStartSLOduration=4.159121286 podStartE2EDuration="22.425865236s" totalImagesPullingTime="18.26674395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:38.425778149 +0000 UTC m=+22.886695881" watchObservedRunningTime="2026-09-17 23:11:38.425865236 +0000 UTC m=+22.886782966"
Sep 17 23:11:39.138655 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:39.138502    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:11:39.326733 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:39.326627    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rfhbd" event={"ID":"2600b302-c671-4f21-9817-6bf74a225573","Type":"ContainerStarted","Data":"787ee9c5b0f39f83ffe80a9d6836881c4ce0cdf5186191bda1f0849c58bc5f55"}
Sep 17 23:11:39.329797 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:39.329766    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl" event={"ID":"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9","Type":"ContainerStarted","Data":"07955bab39d6dd996b565e1ab898fad8df8ec0cf9c987a01a77009f7f0eb9150"}
Sep 17 23:11:39.344983 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:39.344945    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-scwvj" podStartSLOduration=5.071530341 podStartE2EDuration="23.344930738s" totalImagesPullingTime="18.273400397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:38.450921532 +0000 UTC m=+22.911839263" watchObservedRunningTime="2026-09-17 23:11:39.344930738 +0000 UTC m=+23.805848468"
Sep 17 23:11:39.345247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:39.345212    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rfhbd" podStartSLOduration=5.147686301 podStartE2EDuration="23.345202174s" totalImagesPullingTime="18.197515873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:39.344661501 +0000 UTC m=+23.805579233" watchObservedRunningTime="2026-09-17 23:11:39.345202174 +0000 UTC m=+23.806119907"
Sep 17 23:11:40.036979 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.036874    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:11:39.138651851Z","UUID":"1a226248-fb5d-4c83-b50a-818ed6a2c077","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:11:40.038614 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.038590    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:11:40.038614 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.038631    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:11:40.053404 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.053377    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:40.053599 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.053403    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:40.053599 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:40.053495    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:40.053599 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.053377    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:40.053599 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:40.053584    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:40.053754 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:40.053661    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:40.337115 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.337084    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"efd9b794eeeb9aad7c9a1bd1d992959e8829d1dfe0dd319b30819b9ddc0b5f93"}
Sep 17 23:11:40.340346 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.339973    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl" event={"ID":"0acc5aa5-5b54-47fd-8c86-c9df335bcfb9","Type":"ContainerStarted","Data":"dca7653ec74200a82fb7b7b3787fd5726294d8d5054897b015037298c3f3ebea"}
Sep 17 23:11:40.360473 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:40.360420    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dxvgl" podStartSLOduration=4.007314937 podStartE2EDuration="24.36031462s" totalImagesPullingTime="20.352999683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:40.360156465 +0000 UTC m=+24.821074198" watchObservedRunningTime="2026-09-17 23:11:40.36031462 +0000 UTC m=+24.821232350"
Sep 17 23:11:42.053460 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:42.053430    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:42.054048 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:42.053430    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:42.054048 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:42.053573    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:42.054048 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:42.053590    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:42.054048 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:42.053675    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:42.054048 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:42.053746    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:42.360958 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:42.360930    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:42.622884 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:42.622778    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:43.213856 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.213636    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:43.214352 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.214129    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:43.350801 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.350772    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" event={"ID":"4d6e87a1-b303-4d53-ac3e-49acdca61c0d","Type":"ContainerStarted","Data":"9e54b316ddbe5feab106b82d78d68624cd9e553439e69ce42d99c7af7667f1af"}
Sep 17 23:11:43.351045 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.351017    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:43.351115 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.351092    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:43.352976 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.352951    2571 generic.go:356] "Generic (PLEG): container finished" podID="bb310836-ff74-44fd-9ed2-2c624875cc9b" containerID="5097d7c2050f4c8679999f4a9318a4e555f52d6ccd2dbbbc4e3a8cc326db1cd0" exitCode=0
Sep 17 23:11:43.353090 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.353016    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xjllt" event={"ID":"bb310836-ff74-44fd-9ed2-2c624875cc9b","Type":"ContainerDied","Data":"5097d7c2050f4c8679999f4a9318a4e555f52d6ccd2dbbbc4e3a8cc326db1cd0"}
Sep 17 23:11:43.353090 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.353085    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:43.353569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.353549    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-629tc"
Sep 17 23:11:43.367718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.367695    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:43.377143 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.377106    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7" podStartSLOduration=8.834159259 podStartE2EDuration="27.377092255s" totalImagesPullingTime="18.542932996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:43.376600092 +0000 UTC m=+27.837517822" watchObservedRunningTime="2026-09-17 23:11:43.377092255 +0000 UTC m=+27.838009985"
Sep 17 23:11:43.484094 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.484017    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:43.590661 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:43.590640    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:44.055398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.055372    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:44.055398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.055388    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:44.055646 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.055406    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:44.055646 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:44.055489    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:44.056064 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:44.055872    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:44.056064 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:44.055977    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:44.355763 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.355741    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:44.369612 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.369592    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:11:44.759902 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.759618    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-46w55"]
Sep 17 23:11:44.760046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.759917    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:44.760046 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:44.760008    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:44.762682 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.762660    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b6w4t"]
Sep 17 23:11:44.762794 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.762733    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:44.762834 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:44.762802    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:44.763206 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.763187    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jfgnd"]
Sep 17 23:11:44.763268 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.763257    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:44.763372 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:44.763346    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:44.860968 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:44.860940    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.094663 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:45.094628    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.279303 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:45.279260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:45.359968 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:45.359912    2571 generic.go:356] "Generic (PLEG): container finished" podID="bb310836-ff74-44fd-9ed2-2c624875cc9b" containerID="c55072e39c621ecd751013ebefd095249d30584ec71cf1086576a0c2077aa745" exitCode=0
Sep 17 23:11:45.360380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:45.359987    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xjllt" event={"ID":"bb310836-ff74-44fd-9ed2-2c624875cc9b","Type":"ContainerDied","Data":"c55072e39c621ecd751013ebefd095249d30584ec71cf1086576a0c2077aa745"}
Sep 17 23:11:45.839910 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:45.839881    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:46.026247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:46.026217    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:46.055657 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:46.055632    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:46.055807 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:46.055632    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:46.055807 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:46.055730    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:46.055807 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:46.055718    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:46.055925 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:46.055850    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:46.055976 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:46.055931    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:46.234924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:46.234852    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:46.860595 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:46.860568    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.000516 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:47.000490    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.366851 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:47.366816    2571 generic.go:356] "Generic (PLEG): container finished" podID="bb310836-ff74-44fd-9ed2-2c624875cc9b" containerID="b5a18e0b8b55a210e72a9cd19d3e3198d33303cb34958dc391831c3b3f12c64c" exitCode=0
Sep 17 23:11:47.366976 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:47.366855    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xjllt" event={"ID":"bb310836-ff74-44fd-9ed2-2c624875cc9b","Type":"ContainerDied","Data":"b5a18e0b8b55a210e72a9cd19d3e3198d33303cb34958dc391831c3b3f12c64c"}
Sep 17 23:11:47.505814 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:47.505791    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:47.646810 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:47.646741    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:48.053514 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:48.053435    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:48.053934 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:48.053435    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:48.053934 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:48.053562    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:48.053934 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:48.053657    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:48.053934 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:48.053446    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:48.053934 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:48.053761    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:49.719295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:49.719263    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:49.719718 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:49.719434    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:49.719718 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:49.719495    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs podName:243eb096-f63f-4cb7-8e3a-e82e582b400b nodeName:}" failed. No retries permitted until 2026-09-17 23:12:21.719481602 +0000 UTC m=+66.180399315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs") pod "network-metrics-daemon-jfgnd" (UID: "243eb096-f63f-4cb7-8e3a-e82e582b400b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:49.802814 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:49.802782    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:49.819715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:49.819692    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:49.819818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:49.819725    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:49.819854 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:49.819828    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:49.819854 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:49.819832    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:49.819854 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:49.819847    2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:49.819931 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:49.819855    2571 projected.go:196] Error preparing data for projected volume kube-api-access-frg99 for pod openshift-network-diagnostics/network-check-target-b6w4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:49.819931 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:49.819879    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret podName:85f4d3fc-3215-4e00-b5e1-175750c70026 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:21.819866345 +0000 UTC m=+66.280784053 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret") pod "global-pull-secret-syncer-46w55" (UID: "85f4d3fc-3215-4e00-b5e1-175750c70026") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:49.819931 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:49.819897    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99 podName:960a9634-3ae7-41f4-83be-a9b9f3d008a7 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:21.819886652 +0000 UTC m=+66.280804362 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-frg99" (UniqueName: "kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99") pod "network-check-target-b6w4t" (UID: "960a9634-3ae7-41f4-83be-a9b9f3d008a7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:50.053766 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.053689    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:50.053766 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.053714    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:50.053967 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.053694    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:50.053967 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:50.053823    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jfgnd" podUID="243eb096-f63f-4cb7-8e3a-e82e582b400b"
Sep 17 23:11:50.053967 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:50.053915    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-46w55" podUID="85f4d3fc-3215-4e00-b5e1-175750c70026"
Sep 17 23:11:50.054095 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:50.053981    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6w4t" podUID="960a9634-3ae7-41f4-83be-a9b9f3d008a7"
Sep 17 23:11:50.306813 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.306752    2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-246.ec2.internal" event="NodeReady"
Sep 17 23:11:50.306928 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.306876    2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:11:50.351610 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.351583    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vv4bs"]
Sep 17 23:11:50.372156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.372124    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hcfxw"]
Sep 17 23:11:50.372293 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.372244    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.374933 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.374912    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p4tt\""
Sep 17 23:11:50.375082 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.375066    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:11:50.375082 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.375075    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:11:50.384145 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.384126    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vv4bs"]
Sep 17 23:11:50.384242 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.384166    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hcfxw"]
Sep 17 23:11:50.384242 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.384221    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hcfxw"
Sep 17 23:11:50.386698 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.386672    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:11:50.387561 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.387160    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:11:50.387561 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.387182    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pzsn2\""
Sep 17 23:11:50.387561 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.387488    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:11:50.523876 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.523840    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66d22ebd-0199-4fea-a71f-539cafbf0f3d-cert\") pod \"ingress-canary-hcfxw\" (UID: \"66d22ebd-0199-4fea-a71f-539cafbf0f3d\") " pod="openshift-ingress-canary/ingress-canary-hcfxw"
Sep 17 23:11:50.523990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.523885    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42xjw\" (UniqueName: \"kubernetes.io/projected/66d22ebd-0199-4fea-a71f-539cafbf0f3d-kube-api-access-42xjw\") pod \"ingress-canary-hcfxw\" (UID: \"66d22ebd-0199-4fea-a71f-539cafbf0f3d\") " pod="openshift-ingress-canary/ingress-canary-hcfxw"
Sep 17 23:11:50.523990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.523915    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4w8r\" (UniqueName: \"kubernetes.io/projected/243d5166-07d6-4f35-8e7a-16b00a2961e9-kube-api-access-t4w8r\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.523990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.523975    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243d5166-07d6-4f35-8e7a-16b00a2961e9-tmp-dir\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.524131 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.523998    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.524131 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.524015    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/243d5166-07d6-4f35-8e7a-16b00a2961e9-config-volume\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.573409 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.573243    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.624587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.624553    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66d22ebd-0199-4fea-a71f-539cafbf0f3d-cert\") pod \"ingress-canary-hcfxw\" (UID: \"66d22ebd-0199-4fea-a71f-539cafbf0f3d\") " pod="openshift-ingress-canary/ingress-canary-hcfxw"
Sep 17 23:11:50.624699 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.624588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42xjw\" (UniqueName: \"kubernetes.io/projected/66d22ebd-0199-4fea-a71f-539cafbf0f3d-kube-api-access-42xjw\") pod \"ingress-canary-hcfxw\" (UID: \"66d22ebd-0199-4fea-a71f-539cafbf0f3d\") " pod="openshift-ingress-canary/ingress-canary-hcfxw"
Sep 17 23:11:50.624699 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.624616    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4w8r\" (UniqueName: \"kubernetes.io/projected/243d5166-07d6-4f35-8e7a-16b00a2961e9-kube-api-access-t4w8r\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.624699 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.624645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243d5166-07d6-4f35-8e7a-16b00a2961e9-tmp-dir\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.624699 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.624676    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.624880 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.624700    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/243d5166-07d6-4f35-8e7a-16b00a2961e9-config-volume\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.624880 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:50.624859    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:50.624954 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:50.624945    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls podName:243d5166-07d6-4f35-8e7a-16b00a2961e9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:51.124923768 +0000 UTC m=+35.585841494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls") pod "dns-default-vv4bs" (UID: "243d5166-07d6-4f35-8e7a-16b00a2961e9") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:50.625242 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.625222    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/243d5166-07d6-4f35-8e7a-16b00a2961e9-tmp-dir\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.625406 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.625387    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/243d5166-07d6-4f35-8e7a-16b00a2961e9-config-volume\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.629792 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.629635    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66d22ebd-0199-4fea-a71f-539cafbf0f3d-cert\") pod \"ingress-canary-hcfxw\" (UID: \"66d22ebd-0199-4fea-a71f-539cafbf0f3d\") " pod="openshift-ingress-canary/ingress-canary-hcfxw"
Sep 17 23:11:50.634153 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.634134    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4w8r\" (UniqueName: \"kubernetes.io/projected/243d5166-07d6-4f35-8e7a-16b00a2961e9-kube-api-access-t4w8r\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:50.634242 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.634184    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42xjw\" (UniqueName: \"kubernetes.io/projected/66d22ebd-0199-4fea-a71f-539cafbf0f3d-kube-api-access-42xjw\") pod \"ingress-canary-hcfxw\" (UID: \"66d22ebd-0199-4fea-a71f-539cafbf0f3d\") " pod="openshift-ingress-canary/ingress-canary-hcfxw"
Sep 17 23:11:50.688464 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.688437    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:50.693074 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:50.693056    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hcfxw"
Sep 17 23:11:51.127699 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:51.127664    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:51.128102 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:51.127822    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:51.128102 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:51.127883    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls podName:243d5166-07d6-4f35-8e7a-16b00a2961e9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:52.12786868 +0000 UTC m=+36.588786389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls") pod "dns-default-vv4bs" (UID: "243d5166-07d6-4f35-8e7a-16b00a2961e9") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:51.724299 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:51.724238    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hcfxw"]
Sep 17 23:11:51.728443 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:11:51.728414    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d22ebd_0199_4fea_a71f_539cafbf0f3d.slice/crio-ad206293fda287fb71006a2a27bf44302d6e80dd7011685ddae61075edc9fa94 WatchSource:0}: Error finding container ad206293fda287fb71006a2a27bf44302d6e80dd7011685ddae61075edc9fa94: Status 404 returned error can't find the container with id ad206293fda287fb71006a2a27bf44302d6e80dd7011685ddae61075edc9fa94
Sep 17 23:11:52.053548 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.053520    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:11:52.053659 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.053601    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:11:52.053892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.053754    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:11:52.056534 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.056514    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:52.057753 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.057735    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:52.057753 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.057747    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:52.057892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.057735    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dcj7l\""
Sep 17 23:11:52.057892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.057768    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:52.057892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.057737    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zzvsp\""
Sep 17 23:11:52.134891 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.134871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:52.135634 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:52.135017    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:52.135634 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:52.135082    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls podName:243d5166-07d6-4f35-8e7a-16b00a2961e9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:54.135061068 +0000 UTC m=+38.595978795 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls") pod "dns-default-vv4bs" (UID: "243d5166-07d6-4f35-8e7a-16b00a2961e9") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:52.380812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.380774    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hcfxw" event={"ID":"66d22ebd-0199-4fea-a71f-539cafbf0f3d","Type":"ContainerStarted","Data":"ad206293fda287fb71006a2a27bf44302d6e80dd7011685ddae61075edc9fa94"}
Sep 17 23:11:52.385178 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.385151    2571 generic.go:356] "Generic (PLEG): container finished" podID="bb310836-ff74-44fd-9ed2-2c624875cc9b" containerID="df79f2e780d6637b82b91550d87d8612e3c901d68d280bdbd5c4fbe02dd68117" exitCode=0
Sep 17 23:11:52.385312 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:52.385194    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xjllt" event={"ID":"bb310836-ff74-44fd-9ed2-2c624875cc9b","Type":"ContainerDied","Data":"df79f2e780d6637b82b91550d87d8612e3c901d68d280bdbd5c4fbe02dd68117"}
Sep 17 23:11:53.391410 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:53.391372    2571 generic.go:356] "Generic (PLEG): container finished" podID="bb310836-ff74-44fd-9ed2-2c624875cc9b" containerID="b13c1ea84f7ccd740fe381e76de08390cfad415301e8b9dccd332cb5bc0dd64a" exitCode=0
Sep 17 23:11:53.391803 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:53.391437    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xjllt" event={"ID":"bb310836-ff74-44fd-9ed2-2c624875cc9b","Type":"ContainerDied","Data":"b13c1ea84f7ccd740fe381e76de08390cfad415301e8b9dccd332cb5bc0dd64a"}
Sep 17 23:11:54.151562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.151379    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:54.151719 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:54.151517    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:54.151719 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:54.151653    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls podName:243d5166-07d6-4f35-8e7a-16b00a2961e9 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:58.151640138 +0000 UTC m=+42.612557851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls") pod "dns-default-vv4bs" (UID: "243d5166-07d6-4f35-8e7a-16b00a2961e9") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:54.397420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.397389    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hcfxw" event={"ID":"66d22ebd-0199-4fea-a71f-539cafbf0f3d","Type":"ContainerStarted","Data":"1bea9424b41090719f3839a09ce6988f041f92a836d2f545ffe933ae27fbdbc9"}
Sep 17 23:11:54.401152 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.401129    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xjllt" event={"ID":"bb310836-ff74-44fd-9ed2-2c624875cc9b","Type":"ContainerStarted","Data":"31467724c78efaf9bd9587da7c9cc989c63e45f4a76aa6cd6ed0c05e897816bd"}
Sep 17 23:11:54.413898 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.413830    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hcfxw" podStartSLOduration=2.293133747 podStartE2EDuration="4.413815767s" totalImagesPullingTime="2.12068202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:54.412910534 +0000 UTC m=+38.873828264" watchObservedRunningTime="2026-09-17 23:11:54.413815767 +0000 UTC m=+38.874733497"
Sep 17 23:11:54.433859 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.433818    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xjllt" podStartSLOduration=9.129207422 podStartE2EDuration="38.433803435s" totalImagesPullingTime="29.304596013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:54.432573074 +0000 UTC m=+38.893490805" watchObservedRunningTime="2026-09-17 23:11:54.433803435 +0000 UTC m=+38.894721165"
Sep 17 23:11:54.556124 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.556100    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:54.640010 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.639972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:54.750553 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.750486    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:54.855605 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:54.855571    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:57.701981 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:57.701930    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:57.828550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:57.828518    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:11:58.178882 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:11:58.178860    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:11:58.179015 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:58.178962    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:58.179054 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:11:58.179018    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls podName:243d5166-07d6-4f35-8e7a-16b00a2961e9 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:06.179001277 +0000 UTC m=+50.639919004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls") pod "dns-default-vv4bs" (UID: "243d5166-07d6-4f35-8e7a-16b00a2961e9") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:01.954184 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:01.954146    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:02.204966 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:02.204888    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:02.633428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:02.633398    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:02.733924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:02.733893    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:06.228173 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:06.228138    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:12:06.228593 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:06.228281    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:12:06.228593 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:06.228355    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls podName:243d5166-07d6-4f35-8e7a-16b00a2961e9 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:22.22833919 +0000 UTC m=+66.689256898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls") pod "dns-default-vv4bs" (UID: "243d5166-07d6-4f35-8e7a-16b00a2961e9") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:06.977793 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:06.977758    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:07.145594 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:07.145570    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:11.839206 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:11.839163    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:13.117418 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:13.117364    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.380155 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:16.380129    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jsdb7"
Sep 17 23:12:16.533638 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:16.533607    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:16.635722 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:16.635657    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:21.735157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.735118    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:12:21.738167 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.738144    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:12:21.746390 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:21.746371    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:12:21.746465 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:21.746456    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs podName:243eb096-f63f-4cb7-8e3a-e82e582b400b nodeName:}" failed. No retries permitted until 2026-09-17 23:13:25.746433778 +0000 UTC m=+130.207351488 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs") pod "network-metrics-daemon-jfgnd" (UID: "243eb096-f63f-4cb7-8e3a-e82e582b400b") : secret "metrics-daemon-secret" not found
Sep 17 23:12:21.835541 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.835509    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:12:21.835690 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.835558    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:12:21.838577 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.838560    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:12:21.838649 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.838595    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:12:21.848033 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.848005    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:12:21.849551 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.849533    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/85f4d3fc-3215-4e00-b5e1-175750c70026-original-pull-secret\") pod \"global-pull-secret-syncer-46w55\" (UID: \"85f4d3fc-3215-4e00-b5e1-175750c70026\") " pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:12:21.858565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:21.858541    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frg99\" (UniqueName: \"kubernetes.io/projected/960a9634-3ae7-41f4-83be-a9b9f3d008a7-kube-api-access-frg99\") pod \"network-check-target-b6w4t\" (UID: \"960a9634-3ae7-41f4-83be-a9b9f3d008a7\") " pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:12:22.066909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:22.066857    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dcj7l\""
Sep 17 23:12:22.074055 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:22.074040    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-46w55"
Sep 17 23:12:22.075117 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:22.075104    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:12:22.238692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:22.238667    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:12:22.238808 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:22.238763    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:12:22.238850 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:22.238816    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls podName:243d5166-07d6-4f35-8e7a-16b00a2961e9 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:54.238802499 +0000 UTC m=+98.699720208 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls") pod "dns-default-vv4bs" (UID: "243d5166-07d6-4f35-8e7a-16b00a2961e9") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:22.359308 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:22.359282    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b6w4t"]
Sep 17 23:12:22.362034 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:22.362010    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-46w55"]
Sep 17 23:12:22.363079 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:12:22.363057    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960a9634_3ae7_41f4_83be_a9b9f3d008a7.slice/crio-9a8470c78a8380f5eac63134622ca07a70c35e0136573f19aecd9fe5c5418ac3 WatchSource:0}: Error finding container 9a8470c78a8380f5eac63134622ca07a70c35e0136573f19aecd9fe5c5418ac3: Status 404 returned error can't find the container with id 9a8470c78a8380f5eac63134622ca07a70c35e0136573f19aecd9fe5c5418ac3
Sep 17 23:12:22.365786 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:12:22.365740    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f4d3fc_3215_4e00_b5e1_175750c70026.slice/crio-fffb9e12aaffddea6ff69bce902398ca00973c5af87b880d8d7ff0206abb7308 WatchSource:0}: Error finding container fffb9e12aaffddea6ff69bce902398ca00973c5af87b880d8d7ff0206abb7308: Status 404 returned error can't find the container with id fffb9e12aaffddea6ff69bce902398ca00973c5af87b880d8d7ff0206abb7308
Sep 17 23:12:22.465903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:22.465875    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b6w4t" event={"ID":"960a9634-3ae7-41f4-83be-a9b9f3d008a7","Type":"ContainerStarted","Data":"9a8470c78a8380f5eac63134622ca07a70c35e0136573f19aecd9fe5c5418ac3"}
Sep 17 23:12:22.466990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:22.466972    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-46w55" event={"ID":"85f4d3fc-3215-4e00-b5e1-175750c70026","Type":"ContainerStarted","Data":"fffb9e12aaffddea6ff69bce902398ca00973c5af87b880d8d7ff0206abb7308"}
Sep 17 23:12:23.898826 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:23.898770    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:23.986996 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:23.986949    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:27.482182 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:27.482138    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b6w4t" event={"ID":"960a9634-3ae7-41f4-83be-a9b9f3d008a7","Type":"ContainerStarted","Data":"e181d7f6770d6168c00e925b0d648809d867359c7eeccfbec18bacbb7122721c"}
Sep 17 23:12:27.482739 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:27.482257    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:12:27.484117 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:27.484088    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-46w55" event={"ID":"85f4d3fc-3215-4e00-b5e1-175750c70026","Type":"ContainerStarted","Data":"8b63dc4ffa448884d16ead84a04f361ddb1bc7a65e53296671758ce5bf38b9c2"}
Sep 17 23:12:27.517849 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:27.517793    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b6w4t" podStartSLOduration=66.992049929 podStartE2EDuration="1m11.517775528s" totalImagesPullingTime="4.525725599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:27.517002435 +0000 UTC m=+71.977920205" watchObservedRunningTime="2026-09-17 23:12:27.517775528 +0000 UTC m=+71.978693260"
Sep 17 23:12:27.543380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:27.543304    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-46w55" podStartSLOduration=67.016726966 podStartE2EDuration="1m11.543286769s" totalImagesPullingTime="4.526559803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:27.543024263 +0000 UTC m=+72.003941994" watchObservedRunningTime="2026-09-17 23:12:27.543286769 +0000 UTC m=+72.004204501"
Sep 17 23:12:45.159526 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.159400    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"]
Sep 17 23:12:45.161938 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.161919    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.165000 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.164973    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:12:45.167917 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.167890    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7pnxb\""
Sep 17 23:12:45.168031 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.167911    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:12:45.168031 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.167970    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 23:12:45.168031 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.167911    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 23:12:45.172309 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.172287    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"]
Sep 17 23:12:45.256537 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.256509    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"]
Sep 17 23:12:45.258244 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.258228    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.259969 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.259943    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-9w8fn"]
Sep 17 23:12:45.261439 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.261416    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 23:12:45.261547 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.261457    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zjkr5\""
Sep 17 23:12:45.261767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.261747    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.261903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.261751    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 23:12:45.262280 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.262249    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:45.262280 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.262262    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 23:12:45.264422 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.264377    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cc4b8\""
Sep 17 23:12:45.265573 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.265553    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 23:12:45.266116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.266093    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:45.266367 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.266217    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 23:12:45.266732 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.266709    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 23:12:45.271043 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.271021    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"]
Sep 17 23:12:45.272850 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.272828    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 23:12:45.276378 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.276356    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-9w8fn"]
Sep 17 23:12:45.297482 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.297449    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.297482 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.297481    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ph46\" (UniqueName: \"kubernetes.io/projected/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-kube-api-access-9ph46\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.297631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.297503    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74ed121c-9dd7-441c-b9a5-5e902465c1c0-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.297631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.297544    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8glf8\" (UniqueName: \"kubernetes.io/projected/74ed121c-9dd7-441c-b9a5-5e902465c1c0-kube-api-access-8glf8\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.297631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.297563    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.297631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.297592    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74ed121c-9dd7-441c-b9a5-5e902465c1c0-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.377946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.377928    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7bff7b87f5-2qjbd"]
Sep 17 23:12:45.379592 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.379572    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bff7b87f5-2qjbd"]
Sep 17 23:12:45.379685 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.379660    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.382279 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.382260    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:12:45.382408 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.382382    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dcs8c\""
Sep 17 23:12:45.382472 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.382385    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:12:45.382659 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.382643    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:12:45.391441 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.391424    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:12:45.398837 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.398821    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.398903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.398844    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ph46\" (UniqueName: \"kubernetes.io/projected/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-kube-api-access-9ph46\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.398903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.398863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b2bd290-c7f4-4784-a732-0cb7a3090b56-trusted-ca\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.398903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.398883    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74ed121c-9dd7-441c-b9a5-5e902465c1c0-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.399052 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.398913    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8glf8\" (UniqueName: \"kubernetes.io/projected/74ed121c-9dd7-441c-b9a5-5e902465c1c0-kube-api-access-8glf8\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.399052 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:45.398999    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:45.399052 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.399019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b2bd290-c7f4-4784-a732-0cb7a3090b56-config\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.399196 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.399051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljk2n\" (UniqueName: \"kubernetes.io/projected/3b2bd290-c7f4-4784-a732-0cb7a3090b56-kube-api-access-ljk2n\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.399196 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:45.399063    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls podName:74fd4d81-3eb3-43d0-9944-ddc90a0fe735 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:45.899045051 +0000 UTC m=+90.359962766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n5kx8" (UID: "74fd4d81-3eb3-43d0-9944-ddc90a0fe735") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:45.399196 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.399097    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b2bd290-c7f4-4784-a732-0cb7a3090b56-serving-cert\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.399196 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.399124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.399196 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.399159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74ed121c-9dd7-441c-b9a5-5e902465c1c0-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.399722 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.399706    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74ed121c-9dd7-441c-b9a5-5e902465c1c0-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.399758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.399720    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.401193 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.401176    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/74ed121c-9dd7-441c-b9a5-5e902465c1c0-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.412859 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.412807    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8glf8\" (UniqueName: \"kubernetes.io/projected/74ed121c-9dd7-441c-b9a5-5e902465c1c0-kube-api-access-8glf8\") pod \"kube-storage-version-migrator-operator-55889f7b47-q89mj\" (UID: \"74ed121c-9dd7-441c-b9a5-5e902465c1c0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.414912 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.414889    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ph46\" (UniqueName: \"kubernetes.io/projected/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-kube-api-access-9ph46\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.485424 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.485399    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:45.499745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499724    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw6vq\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-kube-api-access-pw6vq\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.499820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499764    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b2bd290-c7f4-4784-a732-0cb7a3090b56-config\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.499820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljk2n\" (UniqueName: \"kubernetes.io/projected/3b2bd290-c7f4-4784-a732-0cb7a3090b56-kube-api-access-ljk2n\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.499890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b2bd290-c7f4-4784-a732-0cb7a3090b56-serving-cert\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.499890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499842    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-trusted-ca\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.499890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499865    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-installation-pull-secrets\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.499890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-bound-sa-token\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.500036 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499945    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e19a0639-aa9e-45d5-8c00-3679618e9abb-ca-trust-extracted\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.500036 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.499996    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-certificates\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.500036 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.500033    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b2bd290-c7f4-4784-a732-0cb7a3090b56-trusted-ca\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.500156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.500050    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-image-registry-private-configuration\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.500156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.500068    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.500382 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.500365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b2bd290-c7f4-4784-a732-0cb7a3090b56-config\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.500680 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.500661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b2bd290-c7f4-4784-a732-0cb7a3090b56-trusted-ca\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.501937 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.501921    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b2bd290-c7f4-4784-a732-0cb7a3090b56-serving-cert\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.519016 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.518984    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljk2n\" (UniqueName: \"kubernetes.io/projected/3b2bd290-c7f4-4784-a732-0cb7a3090b56-kube-api-access-ljk2n\") pod \"console-operator-fd4bb46c-9w8fn\" (UID: \"3b2bd290-c7f4-4784-a732-0cb7a3090b56\") " pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.570458 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.570433    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"
Sep 17 23:12:45.575785 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.575770    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:45.591440 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.591413    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:45.601288 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601258    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-certificates\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.601407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601342    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-image-registry-private-configuration\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.601407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601374    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.601512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601408    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw6vq\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-kube-api-access-pw6vq\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.601512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601469    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-trusted-ca\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.601512 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:45.601488    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:45.601512 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:45.601508    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff7b87f5-2qjbd: secret "image-registry-tls" not found
Sep 17 23:12:45.601751 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:45.601568    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls podName:e19a0639-aa9e-45d5-8c00-3679618e9abb nodeName:}" failed. No retries permitted until 2026-09-17 23:12:46.101547778 +0000 UTC m=+90.562465497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls") pod "image-registry-7bff7b87f5-2qjbd" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb") : secret "image-registry-tls" not found
Sep 17 23:12:45.601824 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601492    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-installation-pull-secrets\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.601824 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601807    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-bound-sa-token\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.601929 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601849    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e19a0639-aa9e-45d5-8c00-3679618e9abb-ca-trust-extracted\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.601929 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.601910    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-certificates\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.602231 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.602205    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e19a0639-aa9e-45d5-8c00-3679618e9abb-ca-trust-extracted\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.602660 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.602641    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-trusted-ca\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.604066 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.604045    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-installation-pull-secrets\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.604291 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.604276    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-image-registry-private-configuration\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.612808 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.612777    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-bound-sa-token\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.612909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.612895    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw6vq\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-kube-api-access-pw6vq\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:45.855534 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.855506    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-9w8fn"]
Sep 17 23:12:45.857467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.857444    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj"]
Sep 17 23:12:45.858454 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:12:45.858429    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2bd290_c7f4_4784_a732_0cb7a3090b56.slice/crio-caaa94a9a82b5f7e5fe2c8cc529b0c9725adaf7cef350d8021dbb3de9c5a36d1 WatchSource:0}: Error finding container caaa94a9a82b5f7e5fe2c8cc529b0c9725adaf7cef350d8021dbb3de9c5a36d1: Status 404 returned error can't find the container with id caaa94a9a82b5f7e5fe2c8cc529b0c9725adaf7cef350d8021dbb3de9c5a36d1
Sep 17 23:12:45.860365 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:12:45.860339    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ed121c_9dd7_441c_b9a5_5e902465c1c0.slice/crio-42a9b748f5735786fea9dc0cf98467465da9c34999b59e4092e1c5bf6efe7ee3 WatchSource:0}: Error finding container 42a9b748f5735786fea9dc0cf98467465da9c34999b59e4092e1c5bf6efe7ee3: Status 404 returned error can't find the container with id 42a9b748f5735786fea9dc0cf98467465da9c34999b59e4092e1c5bf6efe7ee3
Sep 17 23:12:45.904069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:45.904042    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:45.904197 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:45.904181    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:45.904245 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:45.904236    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls podName:74fd4d81-3eb3-43d0-9944-ddc90a0fe735 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:46.904220887 +0000 UTC m=+91.365138596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n5kx8" (UID: "74fd4d81-3eb3-43d0-9944-ddc90a0fe735") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:46.105294 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:46.105267    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:46.105432 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:46.105411    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:46.105432 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:46.105426    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff7b87f5-2qjbd: secret "image-registry-tls" not found
Sep 17 23:12:46.105573 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:46.105472    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls podName:e19a0639-aa9e-45d5-8c00-3679618e9abb nodeName:}" failed. No retries permitted until 2026-09-17 23:12:47.105459167 +0000 UTC m=+91.566376875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls") pod "image-registry-7bff7b87f5-2qjbd" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb") : secret "image-registry-tls" not found
Sep 17 23:12:46.533050 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:46.532897    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" event={"ID":"3b2bd290-c7f4-4784-a732-0cb7a3090b56","Type":"ContainerStarted","Data":"caaa94a9a82b5f7e5fe2c8cc529b0c9725adaf7cef350d8021dbb3de9c5a36d1"}
Sep 17 23:12:46.534705 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:46.534654    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj" event={"ID":"74ed121c-9dd7-441c-b9a5-5e902465c1c0","Type":"ContainerStarted","Data":"42a9b748f5735786fea9dc0cf98467465da9c34999b59e4092e1c5bf6efe7ee3"}
Sep 17 23:12:46.912447 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:46.912412    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:46.912623 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:46.912545    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:46.912623 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:46.912613    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls podName:74fd4d81-3eb3-43d0-9944-ddc90a0fe735 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:48.912594858 +0000 UTC m=+93.373512567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n5kx8" (UID: "74fd4d81-3eb3-43d0-9944-ddc90a0fe735") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:47.113777 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:47.113743    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:47.113950 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:47.113868    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:47.113950 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:47.113886    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff7b87f5-2qjbd: secret "image-registry-tls" not found
Sep 17 23:12:47.113950 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:47.113952    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls podName:e19a0639-aa9e-45d5-8c00-3679618e9abb nodeName:}" failed. No retries permitted until 2026-09-17 23:12:49.113932728 +0000 UTC m=+93.574850439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls") pod "image-registry-7bff7b87f5-2qjbd" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb") : secret "image-registry-tls" not found
Sep 17 23:12:48.029776 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.029746    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:48.130054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.130020    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:48.308971 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.308941    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:48.542253 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.542204    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/0.log"
Sep 17 23:12:48.542253 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.542241    2571 generic.go:356] "Generic (PLEG): container finished" podID="3b2bd290-c7f4-4784-a732-0cb7a3090b56" containerID="37fd6ab17418dfae60a8b7da8c1042632eac4a28e93aafeca52ba1303796559f" exitCode=255
Sep 17 23:12:48.542426 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.542308    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" event={"ID":"3b2bd290-c7f4-4784-a732-0cb7a3090b56","Type":"ContainerDied","Data":"37fd6ab17418dfae60a8b7da8c1042632eac4a28e93aafeca52ba1303796559f"}
Sep 17 23:12:48.542608 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.542592    2571 scope.go:117] "RemoveContainer" containerID="37fd6ab17418dfae60a8b7da8c1042632eac4a28e93aafeca52ba1303796559f"
Sep 17 23:12:48.543965 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.543938    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj" event={"ID":"74ed121c-9dd7-441c-b9a5-5e902465c1c0","Type":"ContainerStarted","Data":"0aace2f4535be90b0157bea5d33e7774a02ce281555b818ab0f2a3bf55762b05"}
Sep 17 23:12:48.574919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.574869    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj" podStartSLOduration=1.268495093 podStartE2EDuration="3.574849571s" totalImagesPullingTime="2.306354478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:48.574390574 +0000 UTC m=+93.035308304" watchObservedRunningTime="2026-09-17 23:12:48.574849571 +0000 UTC m=+93.035767303"
Sep 17 23:12:48.927304 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:48.927279    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:48.927448 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:48.927429    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:48.927517 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:48.927506    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls podName:74fd4d81-3eb3-43d0-9944-ddc90a0fe735 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:52.927486708 +0000 UTC m=+97.388404416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n5kx8" (UID: "74fd4d81-3eb3-43d0-9944-ddc90a0fe735") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:49.128429 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.128400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:49.128715 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:49.128530    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:49.128715 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:49.128546    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff7b87f5-2qjbd: secret "image-registry-tls" not found
Sep 17 23:12:49.128715 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:49.128590    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls podName:e19a0639-aa9e-45d5-8c00-3679618e9abb nodeName:}" failed. No retries permitted until 2026-09-17 23:12:53.128578191 +0000 UTC m=+97.589495900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls") pod "image-registry-7bff7b87f5-2qjbd" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb") : secret "image-registry-tls" not found
Sep 17 23:12:49.548944 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.548921    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/1.log"
Sep 17 23:12:49.549351 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.549315    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/0.log"
Sep 17 23:12:49.549442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.549371    2571 generic.go:356] "Generic (PLEG): container finished" podID="3b2bd290-c7f4-4784-a732-0cb7a3090b56" containerID="e9d23d500e5f05d761a506b70c57a043c25ae511ce0af34c4022ac30709c09da" exitCode=255
Sep 17 23:12:49.549442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.549402    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" event={"ID":"3b2bd290-c7f4-4784-a732-0cb7a3090b56","Type":"ContainerDied","Data":"e9d23d500e5f05d761a506b70c57a043c25ae511ce0af34c4022ac30709c09da"}
Sep 17 23:12:49.549511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.549472    2571 scope.go:117] "RemoveContainer" containerID="37fd6ab17418dfae60a8b7da8c1042632eac4a28e93aafeca52ba1303796559f"
Sep 17 23:12:49.549862 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.549847    2571 scope.go:117] "RemoveContainer" containerID="e9d23d500e5f05d761a506b70c57a043c25ae511ce0af34c4022ac30709c09da"
Sep 17 23:12:49.550031 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:49.550010    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-9w8fn_openshift-console-operator(3b2bd290-c7f4-4784-a732-0cb7a3090b56)\"" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" podUID="3b2bd290-c7f4-4784-a732-0cb7a3090b56"
Sep 17 23:12:49.595058 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.595031    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:49.692794 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.692769    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:49.999425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:49.999401    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-42tst"]
Sep 17 23:12:50.003439 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.003423    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.008919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.008900    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:12:50.009656 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.009518    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:12:50.011131 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.011108    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:12:50.011274 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.011253    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8qgl2\""
Sep 17 23:12:50.011997 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.011979    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.012200 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.012186    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:12:50.043133 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.043101    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-42tst"]
Sep 17 23:12:50.129458 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.129428    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk"]
Sep 17 23:12:50.129884 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.129862    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.132449 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.132432    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk"
Sep 17 23:12:50.134937 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.134919    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fmtjj\""
Sep 17 23:12:50.135032 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.134965    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 23:12:50.135032 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.134988    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.135032 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.135011    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 23:12:50.135178 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.135011    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp27n\" (UniqueName: \"kubernetes.io/projected/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-api-access-kp27n\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.135178 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.135170    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2679bd6e-6fe4-4354-8264-0bffb199bdbd-data-volume\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.135278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.135257    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.135358 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.135295    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2679bd6e-6fe4-4354-8264-0bffb199bdbd-crio-socket\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.140472 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.140450    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk"]
Sep 17 23:12:50.236541 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.236517    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2679bd6e-6fe4-4354-8264-0bffb199bdbd-crio-socket\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.236673 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.236548    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkbg\" (UniqueName: \"kubernetes.io/projected/b894f5f5-bdbc-4b9d-852f-34566f6321e7-kube-api-access-trkbg\") pod \"migrator-76794cbd6c-8wjfk\" (UID: \"b894f5f5-bdbc-4b9d-852f-34566f6321e7\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk"
Sep 17 23:12:50.236673 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.236589    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.236673 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.236604    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp27n\" (UniqueName: \"kubernetes.io/projected/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-api-access-kp27n\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.236673 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.236651    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2679bd6e-6fe4-4354-8264-0bffb199bdbd-crio-socket\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.236866 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.236671    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2679bd6e-6fe4-4354-8264-0bffb199bdbd-data-volume\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.236866 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.236679    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:50.236866 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.236746    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.236866 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.236765    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:12:50.73674347 +0000 UTC m=+95.197661218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:50.236866 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.236818    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:50.237139 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.236891    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:12:50.7368585 +0000 UTC m=+95.197776224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:50.237139 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.236950    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2679bd6e-6fe4-4354-8264-0bffb199bdbd-data-volume\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.253540 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.253496    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp27n\" (UniqueName: \"kubernetes.io/projected/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-api-access-kp27n\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.256952 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.256927    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.337889 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.337865    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkbg\" (UniqueName: \"kubernetes.io/projected/b894f5f5-bdbc-4b9d-852f-34566f6321e7-kube-api-access-trkbg\") pod \"migrator-76794cbd6c-8wjfk\" (UID: \"b894f5f5-bdbc-4b9d-852f-34566f6321e7\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk"
Sep 17 23:12:50.346825 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.346800    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkbg\" (UniqueName: \"kubernetes.io/projected/b894f5f5-bdbc-4b9d-852f-34566f6321e7-kube-api-access-trkbg\") pod \"migrator-76794cbd6c-8wjfk\" (UID: \"b894f5f5-bdbc-4b9d-852f-34566f6321e7\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk"
Sep 17 23:12:50.356125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.356102    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:50.441929 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.441898    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk"
Sep 17 23:12:50.553265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.553210    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/1.log"
Sep 17 23:12:50.553618 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.553600    2571 scope.go:117] "RemoveContainer" containerID="e9d23d500e5f05d761a506b70c57a043c25ae511ce0af34c4022ac30709c09da"
Sep 17 23:12:50.553777 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.553754    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-9w8fn_openshift-console-operator(3b2bd290-c7f4-4784-a732-0cb7a3090b56)\"" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" podUID="3b2bd290-c7f4-4784-a732-0cb7a3090b56"
Sep 17 23:12:50.709871 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.709838    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk"]
Sep 17 23:12:50.712897 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:12:50.712872    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb894f5f5_bdbc_4b9d_852f_34566f6321e7.slice/crio-2ad59b054230ff2a24df760ef74c2d2c1bdaf9c5cb9a6ba6d1a5ca404da0c58d WatchSource:0}: Error finding container 2ad59b054230ff2a24df760ef74c2d2c1bdaf9c5cb9a6ba6d1a5ca404da0c58d: Status 404 returned error can't find the container with id 2ad59b054230ff2a24df760ef74c2d2c1bdaf9c5cb9a6ba6d1a5ca404da0c58d
Sep 17 23:12:50.741512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.741491    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.741605 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:50.741546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:50.741646 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.741608    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:50.741646 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.741620    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:50.741709 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.741654    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:12:51.741640355 +0000 UTC m=+96.202558064 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:50.741709 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:50.741691    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:12:51.741678049 +0000 UTC m=+96.202595762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:51.557972 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:51.557935    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk" event={"ID":"b894f5f5-bdbc-4b9d-852f-34566f6321e7","Type":"ContainerStarted","Data":"2ad59b054230ff2a24df760ef74c2d2c1bdaf9c5cb9a6ba6d1a5ca404da0c58d"}
Sep 17 23:12:51.750205 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:51.750178    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:51.750294 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:51.750278    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:51.750382 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:51.750352    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:51.750439 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:51.750416    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:12:53.750397113 +0000 UTC m=+98.211314838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:51.750439 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:51.750413    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:51.750545 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:51.750478    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:12:53.750459545 +0000 UTC m=+98.211377275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:51.931023 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:51.930996    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:52.028247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:52.028216    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:52.143423 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:52.143405    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ztfrq_522d8a18-aacb-42d7-847e-add2e6355433/dns-node-resolver/0.log"
Sep 17 23:12:52.566613 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:52.566576    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk" event={"ID":"b894f5f5-bdbc-4b9d-852f-34566f6321e7","Type":"ContainerStarted","Data":"08741b9e8970340752a3c180f254fd08b987eaddec02b17b55c2bca594dd7849"}
Sep 17 23:12:52.567029 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:52.566643    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk" event={"ID":"b894f5f5-bdbc-4b9d-852f-34566f6321e7","Type":"ContainerStarted","Data":"5eb5e1d7ae5b674b568874a7a33c301620a259087124b7d9633b9cb187e4a346"}
Sep 17 23:12:52.587766 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:52.587722    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-8wjfk" podStartSLOduration=1.6084183539999999 podStartE2EDuration="2.587704253s" totalImagesPullingTime="979.285899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:12:52.586099887 +0000 UTC m=+97.047017617" watchObservedRunningTime="2026-09-17 23:12:52.587704253 +0000 UTC m=+97.048621984"
Sep 17 23:12:52.958705 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:52.958681    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:12:52.958857 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:52.958836    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:52.958916 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:52.958903    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls podName:74fd4d81-3eb3-43d0-9944-ddc90a0fe735 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:00.958888346 +0000 UTC m=+105.419806059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n5kx8" (UID: "74fd4d81-3eb3-43d0-9944-ddc90a0fe735") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:12:53.143918 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:53.143891    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-scwvj_e0cba546-5344-448a-bed8-1252738e58e2/node-ca/0.log"
Sep 17 23:12:53.160129 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:53.160101    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:12:53.160246 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:53.160223    2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:53.160246 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:53.160235    2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7bff7b87f5-2qjbd: secret "image-registry-tls" not found
Sep 17 23:12:53.160318 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:53.160280    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls podName:e19a0639-aa9e-45d5-8c00-3679618e9abb nodeName:}" failed. No retries permitted until 2026-09-17 23:13:01.160268045 +0000 UTC m=+105.621185754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls") pod "image-registry-7bff7b87f5-2qjbd" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb") : secret "image-registry-tls" not found
Sep 17 23:12:53.419189 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:53.419155    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:53.742937 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:53.742871    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hcfxw_66d22ebd-0199-4fea-a71f-539cafbf0f3d/serve-healthcheck-canary/0.log"
Sep 17 23:12:53.765110 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:53.765084    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:53.765203 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:53.765140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:53.765261 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:53.765209    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:53.765261 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:53.765228    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:53.765364 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:53.765263    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:12:57.765250652 +0000 UTC m=+102.226168375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:53.765364 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:53.765286    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:12:57.765269787 +0000 UTC m=+102.226187496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:54.268758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:54.268729    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:12:54.268893 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:54.268878    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:12:54.268952 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:54.268940    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls podName:243d5166-07d6-4f35-8e7a-16b00a2961e9 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:58.268926566 +0000 UTC m=+162.729844275 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls") pod "dns-default-vv4bs" (UID: "243d5166-07d6-4f35-8e7a-16b00a2961e9") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:54.381603 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:54.381577    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.482027 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:54.481982    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.645045 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:54.645015    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.749376 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:54.749319    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:54.853770 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:54.853737    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.110992 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:55.110970    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:55.576624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:55.576549    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:55.576745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:55.576642    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:12:55.576978 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:55.576964    2571 scope.go:117] "RemoveContainer" containerID="e9d23d500e5f05d761a506b70c57a043c25ae511ce0af34c4022ac30709c09da"
Sep 17 23:12:55.577145 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:55.577126    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-9w8fn_openshift-console-operator(3b2bd290-c7f4-4784-a732-0cb7a3090b56)\"" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" podUID="3b2bd290-c7f4-4784-a732-0cb7a3090b56"
Sep 17 23:12:56.075809 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:56.075769    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:56.219694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:56.219647    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:56.537341 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:56.537293    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:56.704234 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:56.704207    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:56.907083 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:56.907059    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:57.087207 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:57.087172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:57.793810 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:57.793782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:57.793953 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:57.793855    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:12:57.793953 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:57.793904    2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:12:57.794046 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:57.793963    2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:57.794046 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:57.793977    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:13:05.793956143 +0000 UTC m=+110.254873866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : configmap "kube-rbac-proxy" not found
Sep 17 23:12:57.794046 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:12:57.793998    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls podName:2679bd6e-6fe4-4354-8264-0bffb199bdbd nodeName:}" failed. No retries permitted until 2026-09-17 23:13:05.79398787 +0000 UTC m=+110.254905580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42tst" (UID: "2679bd6e-6fe4-4354-8264-0bffb199bdbd") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:12:57.991612 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:57.991582    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.136174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:58.136155    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.275738 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:58.275705    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.476979 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:58.476910    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.489750 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:58.489730    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b6w4t"
Sep 17 23:12:58.673056 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:58.673028    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:58.867738 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:58.867706    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:59.274087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:59.274027    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:12:59.902783 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:12:59.902754    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.470536 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:00.470497    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.665021 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:00.664989    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.765932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:00.765866    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:00.885569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:00.885542    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:01.017082 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:01.017020    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:13:01.017184 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:01.017164    2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:13:01.017254 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:01.017243    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls podName:74fd4d81-3eb3-43d0-9944-ddc90a0fe735 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:17.017228438 +0000 UTC m=+121.478146146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-n5kx8" (UID: "74fd4d81-3eb3-43d0-9944-ddc90a0fe735") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:13:01.218802 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:01.218770    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:13:01.221342 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:01.221304    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"image-registry-7bff7b87f5-2qjbd\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") " pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:13:01.269484 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:01.269408    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:01.292252 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:01.292234    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:13:01.561904 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:01.561832    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7bff7b87f5-2qjbd"]
Sep 17 23:13:01.565855 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:01.565830    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19a0639_aa9e_45d5_8c00_3679618e9abb.slice/crio-b563bc3ed3f3ce520d165800a343013a38e4de3037988e8c309626940a6b4c61 WatchSource:0}: Error finding container b563bc3ed3f3ce520d165800a343013a38e4de3037988e8c309626940a6b4c61: Status 404 returned error can't find the container with id b563bc3ed3f3ce520d165800a343013a38e4de3037988e8c309626940a6b4c61
Sep 17 23:13:01.599886 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:01.599855    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd" event={"ID":"e19a0639-aa9e-45d5-8c00-3679618e9abb","Type":"ContainerStarted","Data":"b563bc3ed3f3ce520d165800a343013a38e4de3037988e8c309626940a6b4c61"}
Sep 17 23:13:01.872009 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:01.871941    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:02.069346 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:02.069294    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:02.604203 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:02.604173    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd" event={"ID":"e19a0639-aa9e-45d5-8c00-3679618e9abb","Type":"ContainerStarted","Data":"978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450"}
Sep 17 23:13:02.604692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:02.604297    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:13:02.625429 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:02.625384    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd" podStartSLOduration=17.625316141 podStartE2EDuration="17.625316141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:02.624351616 +0000 UTC m=+107.085269341" watchObservedRunningTime="2026-09-17 23:13:02.625316141 +0000 UTC m=+107.086233872"
Sep 17 23:13:02.667849 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:02.667811    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:02.816752 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:02.816720    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:02.913781 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:02.913717    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:03.288762 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:03.288683    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:03.898364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:03.898311    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:04.077992 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:04.077941    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:04.677309 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:04.677265    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:04.875222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:04.875186    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:05.072261 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:05.072175    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:05.754847 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:05.754812    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:05.853908 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:05.853876    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:13:05.854061 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:05.853988    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:13:05.854401 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:05.854384    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2679bd6e-6fe4-4354-8264-0bffb199bdbd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:13:05.856367 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:05.856347    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2679bd6e-6fe4-4354-8264-0bffb199bdbd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42tst\" (UID: \"2679bd6e-6fe4-4354-8264-0bffb199bdbd\") " pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:13:05.912282 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:05.912256    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42tst"
Sep 17 23:13:06.183001 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:06.182972    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-42tst"]
Sep 17 23:13:06.185669 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:06.185639    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2679bd6e_6fe4_4354_8264_0bffb199bdbd.slice/crio-4d6d92231f2c056e11ffc0d886d3a4a93523a2e4fa87895005d4ffbed5c24128 WatchSource:0}: Error finding container 4d6d92231f2c056e11ffc0d886d3a4a93523a2e4fa87895005d4ffbed5c24128: Status 404 returned error can't find the container with id 4d6d92231f2c056e11ffc0d886d3a4a93523a2e4fa87895005d4ffbed5c24128
Sep 17 23:13:06.620344 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:06.620303    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42tst" event={"ID":"2679bd6e-6fe4-4354-8264-0bffb199bdbd","Type":"ContainerStarted","Data":"f4ee1726fb90844c73d712f802dd51af6e5bddf8c065d9c9b4092bf762227bc8"}
Sep 17 23:13:06.620487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:06.620370    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42tst" event={"ID":"2679bd6e-6fe4-4354-8264-0bffb199bdbd","Type":"ContainerStarted","Data":"4d6d92231f2c056e11ffc0d886d3a4a93523a2e4fa87895005d4ffbed5c24128"}
Sep 17 23:13:07.624623 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:07.624589    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42tst" event={"ID":"2679bd6e-6fe4-4354-8264-0bffb199bdbd","Type":"ContainerStarted","Data":"dc901614f3d0b0c83c7e5e6ae573a056e2933528b033c9f01cf925a51f78726d"}
Sep 17 23:13:08.628963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:08.628929    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42tst" event={"ID":"2679bd6e-6fe4-4354-8264-0bffb199bdbd","Type":"ContainerStarted","Data":"6426593965aaa77faf60047f8a86ee99e44b49ffd7f0bb1a105ef122c4564413"}
Sep 17 23:13:08.648737 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:08.648697    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-42tst" podStartSLOduration=17.675470735 podStartE2EDuration="19.648680804s" totalImagesPullingTime="1.973210069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:12:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:08.646699301 +0000 UTC m=+113.107617063" watchObservedRunningTime="2026-09-17 23:13:08.648680804 +0000 UTC m=+113.109598529"
Sep 17 23:13:10.053220 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:10.053190    2571 scope.go:117] "RemoveContainer" containerID="e9d23d500e5f05d761a506b70c57a043c25ae511ce0af34c4022ac30709c09da"
Sep 17 23:13:10.636192 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:10.636166    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:13:10.636605 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:10.636584    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/1.log"
Sep 17 23:13:10.636699 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:10.636628    2571 generic.go:356] "Generic (PLEG): container finished" podID="3b2bd290-c7f4-4784-a732-0cb7a3090b56" containerID="afd0363f62f4bb4d6e32d383af64c1bd3452336c6d9d27b62c7d28e038de4de6" exitCode=255
Sep 17 23:13:10.636752 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:10.636701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" event={"ID":"3b2bd290-c7f4-4784-a732-0cb7a3090b56","Type":"ContainerDied","Data":"afd0363f62f4bb4d6e32d383af64c1bd3452336c6d9d27b62c7d28e038de4de6"}
Sep 17 23:13:10.636806 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:10.636757    2571 scope.go:117] "RemoveContainer" containerID="e9d23d500e5f05d761a506b70c57a043c25ae511ce0af34c4022ac30709c09da"
Sep 17 23:13:10.637057 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:10.637032    2571 scope.go:117] "RemoveContainer" containerID="afd0363f62f4bb4d6e32d383af64c1bd3452336c6d9d27b62c7d28e038de4de6"
Sep 17 23:13:10.637201 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:10.637182    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-9w8fn_openshift-console-operator(3b2bd290-c7f4-4784-a732-0cb7a3090b56)\"" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" podUID="3b2bd290-c7f4-4784-a732-0cb7a3090b56"
Sep 17 23:13:11.641082 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:11.641051    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:13:13.345089 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:13.345045    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:13.445866 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:13.445840    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:14.440069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.440041    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bff7b87f5-2qjbd"]
Sep 17 23:13:14.477824 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.477797    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-565bb49474-ht2vn"]
Sep 17 23:13:14.480775 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.480755    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.502029 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.502000    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565bb49474-ht2vn"]
Sep 17 23:13:14.502509 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.502485    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:14.602122 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.602090    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:14.614129 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.614108    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3698b98e-7c45-4f94-9618-857211a5ccf0-trusted-ca\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.614207 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.614145    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3698b98e-7c45-4f94-9618-857211a5ccf0-registry-certificates\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.614207 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.614162    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3698b98e-7c45-4f94-9618-857211a5ccf0-ca-trust-extracted\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.614207 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.614189    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3698b98e-7c45-4f94-9618-857211a5ccf0-image-registry-private-configuration\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.614357 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.614230    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3698b98e-7c45-4f94-9618-857211a5ccf0-installation-pull-secrets\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.614357 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.614264    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88gj\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-kube-api-access-x88gj\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.614357 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.614302    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-bound-sa-token\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.614357 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.614354    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-registry-tls\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715262    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3698b98e-7c45-4f94-9618-857211a5ccf0-registry-certificates\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715284    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3698b98e-7c45-4f94-9618-857211a5ccf0-ca-trust-extracted\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715312    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3698b98e-7c45-4f94-9618-857211a5ccf0-image-registry-private-configuration\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715351    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3698b98e-7c45-4f94-9618-857211a5ccf0-installation-pull-secrets\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715373    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88gj\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-kube-api-access-x88gj\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715397    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-bound-sa-token\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-registry-tls\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715482    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3698b98e-7c45-4f94-9618-857211a5ccf0-trusted-ca\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.715842 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.715815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3698b98e-7c45-4f94-9618-857211a5ccf0-ca-trust-extracted\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.716117 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.716094    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3698b98e-7c45-4f94-9618-857211a5ccf0-registry-certificates\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.716539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.716522    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3698b98e-7c45-4f94-9618-857211a5ccf0-trusted-ca\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.718005 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.717987    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-registry-tls\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.718391 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.718369    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3698b98e-7c45-4f94-9618-857211a5ccf0-image-registry-private-configuration\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.718468 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.718423    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3698b98e-7c45-4f94-9618-857211a5ccf0-installation-pull-secrets\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.727914 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.727891    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-bound-sa-token\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.728248 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.728233    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88gj\" (UniqueName: \"kubernetes.io/projected/3698b98e-7c45-4f94-9618-857211a5ccf0-kube-api-access-x88gj\") pod \"image-registry-565bb49474-ht2vn\" (UID: \"3698b98e-7c45-4f94-9618-857211a5ccf0\") " pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:14.789470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:14.789451    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:15.076477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:15.076342    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-565bb49474-ht2vn"]
Sep 17 23:13:15.078907 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:15.078881    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3698b98e_7c45_4f94_9618_857211a5ccf0.slice/crio-b5fe6f833d132e94bd8133a5dc22f011bed4bcd0dc234d37401867fdbb101a6b WatchSource:0}: Error finding container b5fe6f833d132e94bd8133a5dc22f011bed4bcd0dc234d37401867fdbb101a6b: Status 404 returned error can't find the container with id b5fe6f833d132e94bd8133a5dc22f011bed4bcd0dc234d37401867fdbb101a6b
Sep 17 23:13:15.576132 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:15.576100    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:13:15.576553 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:15.576152    2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:13:15.576553 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:15.576442    2571 scope.go:117] "RemoveContainer" containerID="afd0363f62f4bb4d6e32d383af64c1bd3452336c6d9d27b62c7d28e038de4de6"
Sep 17 23:13:15.576621 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:15.576596    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-9w8fn_openshift-console-operator(3b2bd290-c7f4-4784-a732-0cb7a3090b56)\"" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" podUID="3b2bd290-c7f4-4784-a732-0cb7a3090b56"
Sep 17 23:13:15.654355 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:15.654308    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565bb49474-ht2vn" event={"ID":"3698b98e-7c45-4f94-9618-857211a5ccf0","Type":"ContainerStarted","Data":"96c25982cdcb194fc10a3d2b00448ffdc7a26efe34e8698012f7f00aa5f64851"}
Sep 17 23:13:15.654485 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:15.654398    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-565bb49474-ht2vn" event={"ID":"3698b98e-7c45-4f94-9618-857211a5ccf0","Type":"ContainerStarted","Data":"b5fe6f833d132e94bd8133a5dc22f011bed4bcd0dc234d37401867fdbb101a6b"}
Sep 17 23:13:15.654485 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:15.654437    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:15.677153 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:15.677115    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-565bb49474-ht2vn" podStartSLOduration=1.677102469 podStartE2EDuration="1.677102469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:15.67651371 +0000 UTC m=+120.137431442" watchObservedRunningTime="2026-09-17 23:13:15.677102469 +0000 UTC m=+120.138020199"
Sep 17 23:13:17.030035 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:17.029990    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:13:17.032432 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:17.032409    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/74fd4d81-3eb3-43d0-9944-ddc90a0fe735-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-n5kx8\" (UID: \"74fd4d81-3eb3-43d0-9944-ddc90a0fe735\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:13:17.206114 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:17.206070    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:17.273952 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:17.273926    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7pnxb\""
Sep 17 23:13:17.282170 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:17.282128    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"
Sep 17 23:13:17.322449 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:17.322423    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:17.553127 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:17.553050    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8"]
Sep 17 23:13:17.556135 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:17.556108    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fd4d81_3eb3_43d0_9944_ddc90a0fe735.slice/crio-6419f0b7e98e63b1343984d1485690abba522174398f0cb9585697d3168349cd WatchSource:0}: Error finding container 6419f0b7e98e63b1343984d1485690abba522174398f0cb9585697d3168349cd: Status 404 returned error can't find the container with id 6419f0b7e98e63b1343984d1485690abba522174398f0cb9585697d3168349cd
Sep 17 23:13:17.660888 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:17.660858    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8" event={"ID":"74fd4d81-3eb3-43d0-9944-ddc90a0fe735","Type":"ContainerStarted","Data":"6419f0b7e98e63b1343984d1485690abba522174398f0cb9585697d3168349cd"}
Sep 17 23:13:19.299646 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:19.299620    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:19.409837 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:19.409791    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:19.669009 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:19.668977    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8" event={"ID":"74fd4d81-3eb3-43d0-9944-ddc90a0fe735","Type":"ContainerStarted","Data":"b9bc63b0e4ff9cb3cb2e84a5d018596f64f2aed9efa97c769d3b90d525af5d2c"}
Sep 17 23:13:19.674439 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:19.674404    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:19.690450 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:19.690409    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-n5kx8" podStartSLOduration=32.953227185 podStartE2EDuration="34.690394429s" totalImagesPullingTime="1.737167244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:19.689658661 +0000 UTC m=+124.150576403" watchObservedRunningTime="2026-09-17 23:13:19.690394429 +0000 UTC m=+124.151312204"
Sep 17 23:13:19.843512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:19.843478    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:19.943084 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:19.943013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:22.828408 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.828369    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:22.864447 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.864420    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"]
Sep 17 23:13:22.867023 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.867008    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:22.869856 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.869838    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 23:13:22.871454 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.871432    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tzzxj\""
Sep 17 23:13:22.871551 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.871432    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 23:13:22.871551 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.871488    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:13:22.877509 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.877491    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"]
Sep 17 23:13:22.912502 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.912472    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:22.970497 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.970471    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4nkl\" (UniqueName: \"kubernetes.io/projected/871db9ce-0962-47a2-86d2-6327063f7661-kube-api-access-v4nkl\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:22.970596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.970506    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/871db9ce-0962-47a2-86d2-6327063f7661-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:22.970596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.970525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/871db9ce-0962-47a2-86d2-6327063f7661-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:22.970707 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.970613    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/871db9ce-0962-47a2-86d2-6327063f7661-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:22.995839 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:22.992848    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:23.071039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.071011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4nkl\" (UniqueName: \"kubernetes.io/projected/871db9ce-0962-47a2-86d2-6327063f7661-kube-api-access-v4nkl\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.071122 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.071048    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/871db9ce-0962-47a2-86d2-6327063f7661-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.071122 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.071064    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/871db9ce-0962-47a2-86d2-6327063f7661-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.071122 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.071088    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/871db9ce-0962-47a2-86d2-6327063f7661-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.071691 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.071674    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/871db9ce-0962-47a2-86d2-6327063f7661-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.073696 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.073671    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/871db9ce-0962-47a2-86d2-6327063f7661-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.073778 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.073757    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/871db9ce-0962-47a2-86d2-6327063f7661-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.082192 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.082147    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4nkl\" (UniqueName: \"kubernetes.io/projected/871db9ce-0962-47a2-86d2-6327063f7661-kube-api-access-v4nkl\") pod \"prometheus-operator-7bd5d56d6-nqc4j\" (UID: \"871db9ce-0962-47a2-86d2-6327063f7661\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.176014 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.175991    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"
Sep 17 23:13:23.442852 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.442828    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j"]
Sep 17 23:13:23.444849 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:23.444822    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871db9ce_0962_47a2_86d2_6327063f7661.slice/crio-b512ce022e5d6eaf3d926859511b6ffc1edefd99af7c2b70e94d4d901547adbb WatchSource:0}: Error finding container b512ce022e5d6eaf3d926859511b6ffc1edefd99af7c2b70e94d4d901547adbb: Status 404 returned error can't find the container with id b512ce022e5d6eaf3d926859511b6ffc1edefd99af7c2b70e94d4d901547adbb
Sep 17 23:13:23.682105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:23.682078    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j" event={"ID":"871db9ce-0962-47a2-86d2-6327063f7661","Type":"ContainerStarted","Data":"b512ce022e5d6eaf3d926859511b6ffc1edefd99af7c2b70e94d4d901547adbb"}
Sep 17 23:13:24.203090 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:24.203053    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:24.446373 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:24.446318    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:13:24.982832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:24.982793    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:25.690665 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:25.690631    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j" event={"ID":"871db9ce-0962-47a2-86d2-6327063f7661","Type":"ContainerStarted","Data":"1627de1d237c97e9e9a0b77834ba9de2d976fbfc55dc7948abf6a389c2acd3f3"}
Sep 17 23:13:25.691037 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:25.690686    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j" event={"ID":"871db9ce-0962-47a2-86d2-6327063f7661","Type":"ContainerStarted","Data":"a74ff71e09a7c9d100820dc4b9ce3024d403f50c6920ded283bcd139b2a581a7"}
Sep 17 23:13:25.711297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:25.711252    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-nqc4j" podStartSLOduration=2.418581089 podStartE2EDuration="3.711236992s" totalImagesPullingTime="1.292655903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:25.710202416 +0000 UTC m=+130.171120160" watchObservedRunningTime="2026-09-17 23:13:25.711236992 +0000 UTC m=+130.172154723"
Sep 17 23:13:25.790763 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:25.790734    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:13:25.793102 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:25.793085    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/243eb096-f63f-4cb7-8e3a-e82e582b400b-metrics-certs\") pod \"network-metrics-daemon-jfgnd\" (UID: \"243eb096-f63f-4cb7-8e3a-e82e582b400b\") " pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:13:25.972596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:25.972523    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zzvsp\""
Sep 17 23:13:25.980688 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:25.980671    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jfgnd"
Sep 17 23:13:26.250914 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:26.250889    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jfgnd"]
Sep 17 23:13:26.253482 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:26.253460    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243eb096_f63f_4cb7_8e3a_e82e582b400b.slice/crio-ceab9f329c503da6ed29584e5b93c0b05a5da97de4deebacce03917ea7a9c82d WatchSource:0}: Error finding container ceab9f329c503da6ed29584e5b93c0b05a5da97de4deebacce03917ea7a9c82d: Status 404 returned error can't find the container with id ceab9f329c503da6ed29584e5b93c0b05a5da97de4deebacce03917ea7a9c82d
Sep 17 23:13:26.695239 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:26.695204    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfgnd" event={"ID":"243eb096-f63f-4cb7-8e3a-e82e582b400b","Type":"ContainerStarted","Data":"ceab9f329c503da6ed29584e5b93c0b05a5da97de4deebacce03917ea7a9c82d"}
Sep 17 23:13:26.958997 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:26.958889    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.056256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.056212    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.196436 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.196397    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.241820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.241798    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4kngq"]
Sep 17 23:13:27.244393 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.244376    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.246862 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.246843    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:13:27.246955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.246935    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4h2zn\""
Sep 17 23:13:27.247116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.247102    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:13:27.247156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.247102    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:13:27.300126 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300099    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-textfile\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.300246 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300143    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qr8\" (UniqueName: \"kubernetes.io/projected/5a8f129a-20d8-4df9-ad0a-8018743ec533-kube-api-access-97qr8\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.300246 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300182    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-root\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.300246 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300206    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.300436 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300237    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.300436 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300407    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-sys\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.300535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-wtmp\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.300589 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300538    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-accelerators-collector-config\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.300589 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.300564    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f129a-20d8-4df9-ad0a-8018743ec533-metrics-client-ca\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.302475 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.302307    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.400933 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.400907    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f129a-20d8-4df9-ad0a-8018743ec533-metrics-client-ca\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.400958    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-textfile\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.400985    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qr8\" (UniqueName: \"kubernetes.io/projected/5a8f129a-20d8-4df9-ad0a-8018743ec533-kube-api-access-97qr8\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-root\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401028    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401058    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401085    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-sys\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401120    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-wtmp\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401148    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-root\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401162    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-accelerators-collector-config\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401415 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:27.401183    2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:13:27.401415 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:27.401243    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls podName:5a8f129a-20d8-4df9-ad0a-8018743ec533 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:27.901223923 +0000 UTC m=+132.362141632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls") pod "node-exporter-4kngq" (UID: "5a8f129a-20d8-4df9-ad0a-8018743ec533") : secret "node-exporter-tls" not found
Sep 17 23:13:27.401415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401259    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-sys\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401379    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-wtmp\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401604    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f129a-20d8-4df9-ad0a-8018743ec533-metrics-client-ca\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401648    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-accelerators-collector-config\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.401745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.401649    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-textfile\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.403644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.403622    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.413959 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.413930    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qr8\" (UniqueName: \"kubernetes.io/projected/5a8f129a-20d8-4df9-ad0a-8018743ec533-kube-api-access-97qr8\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.439938 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.439909    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.540710 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.540621    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.644297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.644262    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.700465 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.700433    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfgnd" event={"ID":"243eb096-f63f-4cb7-8e3a-e82e582b400b","Type":"ContainerStarted","Data":"d8a44e80ddc2ba6b36c8248e005035b6df1fdb0544b96dfed6a68473701f9fe5"}
Sep 17 23:13:27.700791 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.700490    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jfgnd" event={"ID":"243eb096-f63f-4cb7-8e3a-e82e582b400b","Type":"ContainerStarted","Data":"4e94af5ba536d6a776ba39b3a467517000f457301d4082caa98e76ac230d6fd0"}
Sep 17 23:13:27.724007 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.723967    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jfgnd" podStartSLOduration=130.790218746 podStartE2EDuration="2m11.723953721s" totalImagesPullingTime="933.734975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:27.722601341 +0000 UTC m=+132.183519081" watchObservedRunningTime="2026-09-17 23:13:27.723953721 +0000 UTC m=+132.184871484"
Sep 17 23:13:27.741960 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.741937    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.859254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.859227    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:27.905088 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.905063    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:27.905197 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:27.905067    2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:13:27.905197 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:27.905190    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls podName:5a8f129a-20d8-4df9-ad0a-8018743ec533 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:28.905174849 +0000 UTC m=+133.366092562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls") pod "node-exporter-4kngq" (UID: "5a8f129a-20d8-4df9-ad0a-8018743ec533") : secret "node-exporter-tls" not found
Sep 17 23:13:27.962655 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:27.962630    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:28.145927 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:28.145847    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:28.242058 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:28.242026    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:28.329262 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:28.329237    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:28.745449 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:28.745416    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:28.912985 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:28.912957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:28.915304 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:28.915276    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5a8f129a-20d8-4df9-ad0a-8018743ec533-node-exporter-tls\") pod \"node-exporter-4kngq\" (UID: \"5a8f129a-20d8-4df9-ad0a-8018743ec533\") " pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:29.059825 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.059755    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4kngq"
Sep 17 23:13:29.071108 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:29.071085    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8f129a_20d8_4df9_ad0a_8018743ec533.slice/crio-0a599e0f9629134309b9afb5b4ba09c297cb83d08e3780a0e35f525c077f7b3d WatchSource:0}: Error finding container 0a599e0f9629134309b9afb5b4ba09c297cb83d08e3780a0e35f525c077f7b3d: Status 404 returned error can't find the container with id 0a599e0f9629134309b9afb5b4ba09c297cb83d08e3780a0e35f525c077f7b3d
Sep 17 23:13:29.322627 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.322560    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"]
Sep 17 23:13:29.325230 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.325216    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.327757 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.327738    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-et0nr933p65m4\""
Sep 17 23:13:29.328116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.328085    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 23:13:29.328116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.328094    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 23:13:29.328242 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.328145    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 23:13:29.329131 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.329117    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 23:13:29.329189 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.329138    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 23:13:29.329240 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.329138    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-d7667\""
Sep 17 23:13:29.342973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.342952    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"]
Sep 17 23:13:29.416962 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.416940    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.417048 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.416985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-grpc-tls\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.417048 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.417006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.417124 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.417041    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32762502-7f12-4775-9465-1c8f9025a694-metrics-client-ca\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.417124 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.417077    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.417185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.417127    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9c88\" (UniqueName: \"kubernetes.io/projected/32762502-7f12-4775-9465-1c8f9025a694-kube-api-access-q9c88\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.417185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.417158    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-tls\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.417244 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.417188    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.518104 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.518081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.518212 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.518107    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32762502-7f12-4775-9465-1c8f9025a694-metrics-client-ca\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.518212 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.518124    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.518212 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.518146    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9c88\" (UniqueName: \"kubernetes.io/projected/32762502-7f12-4775-9465-1c8f9025a694-kube-api-access-q9c88\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.518212 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.518166    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-tls\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.518438 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.518299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.518438 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.518375    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.518438 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.518432    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-grpc-tls\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.519262 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.519019    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32762502-7f12-4775-9465-1c8f9025a694-metrics-client-ca\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.521149 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.521097    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.521149 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.521097    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.521278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.521257    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-tls\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.521407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.521385    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.521597 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.521576    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.521639 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.521625    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/32762502-7f12-4775-9465-1c8f9025a694-secret-grpc-tls\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.527721 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.527703    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9c88\" (UniqueName: \"kubernetes.io/projected/32762502-7f12-4775-9465-1c8f9025a694-kube-api-access-q9c88\") pod \"thanos-querier-77f5b54f8c-vxsjv\" (UID: \"32762502-7f12-4775-9465-1c8f9025a694\") " pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.634905 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.634880    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:29.709076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.709021    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kngq" event={"ID":"5a8f129a-20d8-4df9-ad0a-8018743ec533","Type":"ContainerStarted","Data":"0a599e0f9629134309b9afb5b4ba09c297cb83d08e3780a0e35f525c077f7b3d"}
Sep 17 23:13:29.920349 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:29.920290    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"]
Sep 17 23:13:30.012435 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:30.012404    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32762502_7f12_4775_9465_1c8f9025a694.slice/crio-a2028dc23c0f98eaeb485f63717a8d4ec36391eb24b592328f009af7475a1c4b WatchSource:0}: Error finding container a2028dc23c0f98eaeb485f63717a8d4ec36391eb24b592328f009af7475a1c4b: Status 404 returned error can't find the container with id a2028dc23c0f98eaeb485f63717a8d4ec36391eb24b592328f009af7475a1c4b
Sep 17 23:13:30.193210 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:30.193186    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:30.319472 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:30.319442    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:30.430350 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:30.430268    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:30.530902 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:30.530872    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:30.719158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:30.719074    2571 generic.go:356] "Generic (PLEG): container finished" podID="5a8f129a-20d8-4df9-ad0a-8018743ec533" containerID="40cbbc322ae2d9bcb8d4ddd0272a4ec8ae573f8d97f9953968f6a9aca8d4ee43" exitCode=0
Sep 17 23:13:30.719277 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:30.719171    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kngq" event={"ID":"5a8f129a-20d8-4df9-ad0a-8018743ec533","Type":"ContainerDied","Data":"40cbbc322ae2d9bcb8d4ddd0272a4ec8ae573f8d97f9953968f6a9aca8d4ee43"}
Sep 17 23:13:30.720443 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:30.720421    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv" event={"ID":"32762502-7f12-4775-9465-1c8f9025a694","Type":"ContainerStarted","Data":"a2028dc23c0f98eaeb485f63717a8d4ec36391eb24b592328f009af7475a1c4b"}
Sep 17 23:13:31.053841 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.053772    2571 scope.go:117] "RemoveContainer" containerID="afd0363f62f4bb4d6e32d383af64c1bd3452336c6d9d27b62c7d28e038de4de6"
Sep 17 23:13:31.727037 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.727005    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:13:31.727213 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.727142    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" event={"ID":"3b2bd290-c7f4-4784-a732-0cb7a3090b56","Type":"ContainerStarted","Data":"db7e98bbca1ee94019408ba8d58e381b68230b4218e1959eef9c216c27f466d9"}
Sep 17 23:13:31.727521 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.727466    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:13:31.730003 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.729956    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kngq" event={"ID":"5a8f129a-20d8-4df9-ad0a-8018743ec533","Type":"ContainerStarted","Data":"d61fc7eed33edab32e6933e1ddc609f713fa20dc2500c47536cafdc527121f6b"}
Sep 17 23:13:31.730140 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.730006    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4kngq" event={"ID":"5a8f129a-20d8-4df9-ad0a-8018743ec533","Type":"ContainerStarted","Data":"31ff407818cc4ff1f81a1229723e348ea4fd23b12bc0de89ee9088795ac8b863"}
Sep 17 23:13:31.733013 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.732990    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn"
Sep 17 23:13:31.747745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.747694    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-9w8fn" podStartSLOduration=44.438091523 podStartE2EDuration="46.747675306s" totalImagesPullingTime="2.309583783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:12:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:31.745404442 +0000 UTC m=+136.206322176" watchObservedRunningTime="2026-09-17 23:13:31.747675306 +0000 UTC m=+136.208593040"
Sep 17 23:13:31.789922 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.789870    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4kngq" podStartSLOduration=3.755826265 podStartE2EDuration="4.789850275s" totalImagesPullingTime="962.698328ms" totalInitContainerRuntime="71.325682ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:31.7894466 +0000 UTC m=+136.250364330" watchObservedRunningTime="2026-09-17 23:13:31.789850275 +0000 UTC m=+136.250768007"
Sep 17 23:13:31.864641 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.864594    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:31.969557 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.969515    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:31.972744 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.972699    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"]
Sep 17 23:13:31.974820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.974802    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:31.977386 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.977285    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-6qlrz\""
Sep 17 23:13:31.977386 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.977300    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 23:13:31.983734 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:31.983712    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"]
Sep 17 23:13:32.040092 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.040068    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10d39d3d-7f7f-460e-b412-09c0280fdfbd-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-fspzn\" (UID: \"10d39d3d-7f7f-460e-b412-09c0280fdfbd\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:32.040230 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.040116    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdb4l\" (UniqueName: \"kubernetes.io/projected/10d39d3d-7f7f-460e-b412-09c0280fdfbd-kube-api-access-gdb4l\") pod \"monitoring-plugin-5984459ff8-fspzn\" (UID: \"10d39d3d-7f7f-460e-b412-09c0280fdfbd\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:32.068199 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.068162    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:32.141385 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.141347    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10d39d3d-7f7f-460e-b412-09c0280fdfbd-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-fspzn\" (UID: \"10d39d3d-7f7f-460e-b412-09c0280fdfbd\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:32.141518 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.141431    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdb4l\" (UniqueName: \"kubernetes.io/projected/10d39d3d-7f7f-460e-b412-09c0280fdfbd-kube-api-access-gdb4l\") pod \"monitoring-plugin-5984459ff8-fspzn\" (UID: \"10d39d3d-7f7f-460e-b412-09c0280fdfbd\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:32.141592 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:32.141535    2571 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 23:13:32.141640 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:32.141623    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10d39d3d-7f7f-460e-b412-09c0280fdfbd-monitoring-plugin-cert podName:10d39d3d-7f7f-460e-b412-09c0280fdfbd nodeName:}" failed. No retries permitted until 2026-09-17 23:13:32.641600353 +0000 UTC m=+137.102518081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/10d39d3d-7f7f-460e-b412-09c0280fdfbd-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-fspzn" (UID: "10d39d3d-7f7f-460e-b412-09c0280fdfbd") : secret "monitoring-plugin-cert" not found
Sep 17 23:13:32.159113 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.159092    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdb4l\" (UniqueName: \"kubernetes.io/projected/10d39d3d-7f7f-460e-b412-09c0280fdfbd-kube-api-access-gdb4l\") pod \"monitoring-plugin-5984459ff8-fspzn\" (UID: \"10d39d3d-7f7f-460e-b412-09c0280fdfbd\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:32.163782 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.163759    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:32.251812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.251668    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:32.645264 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.645231    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10d39d3d-7f7f-460e-b412-09c0280fdfbd-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-fspzn\" (UID: \"10d39d3d-7f7f-460e-b412-09c0280fdfbd\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:32.647516 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.647498    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/10d39d3d-7f7f-460e-b412-09c0280fdfbd-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-fspzn\" (UID: \"10d39d3d-7f7f-460e-b412-09c0280fdfbd\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:32.657778 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.657727    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:32.734982 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.734958    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv" event={"ID":"32762502-7f12-4775-9465-1c8f9025a694","Type":"ContainerStarted","Data":"ea7fd806de769a3272db65212cfdfc42ca6adfe46c857a61bd2db5ace50a7482"}
Sep 17 23:13:32.759484 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.759337    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:32.857881 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.857849    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:32.884664 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.884610    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:32.961715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:32.961610    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:33.154756 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.154735    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"]
Sep 17 23:13:33.157071 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:33.157002    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d39d3d_7f7f_460e_b412_09c0280fdfbd.slice/crio-20d076f5de1259bd261e2f984358ea879f8f7736d1d70a544921cc12400c6648 WatchSource:0}: Error finding container 20d076f5de1259bd261e2f984358ea879f8f7736d1d70a544921cc12400c6648: Status 404 returned error can't find the container with id 20d076f5de1259bd261e2f984358ea879f8f7736d1d70a544921cc12400c6648
Sep 17 23:13:33.168713 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.168693    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:33.314107 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.313944    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:33.420141 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.419981    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:33.514647 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.514613    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:33.556713 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.556674    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:13:33.560396 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.560375    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.563798 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.563775    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:13:33.563931 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.563913    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:13:33.564116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.564098    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:13:33.564315 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.564296    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bsfvr\""
Sep 17 23:13:33.564576 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.564554    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:13:33.564685 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.564606    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:13:33.564685 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.564674    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:13:33.564805 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.564786    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:13:33.564881 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.564815    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:13:33.565511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.565386    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:13:33.565511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.565396    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:13:33.565674 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.565639    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:13:33.566354 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.565734    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:13:33.566354 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.565762    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:13:33.566354 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.565860    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-db3mnal6kjvu5\""
Sep 17 23:13:33.567928 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.567905    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:13:33.572200 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.571394    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:13:33.576065 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.576002    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:13:33.655585 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.655690 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.655690 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655620    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.655825 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655703    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.655825 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655738    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.655825 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.655825 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655819    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7xk6\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-kube-api-access-w7xk6\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656012 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655850    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656012 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655883    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656012 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655911    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656012 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655936    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656012 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655959    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656012 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.655986    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.656034    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.656078    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.656104    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.656132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.656185    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.656208    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.656292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.656255    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.713991 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.713961    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:33.742998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.742974    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn" event={"ID":"10d39d3d-7f7f-460e-b412-09c0280fdfbd","Type":"ContainerStarted","Data":"20d076f5de1259bd261e2f984358ea879f8f7736d1d70a544921cc12400c6648"}
Sep 17 23:13:33.745609 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.745521    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv" event={"ID":"32762502-7f12-4775-9465-1c8f9025a694","Type":"ContainerStarted","Data":"ee28575976afd3a0e48fac0291b9c7b6c678bbd75eb2e3d347ad3af6867162f5"}
Sep 17 23:13:33.745688 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.745671    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv" event={"ID":"32762502-7f12-4775-9465-1c8f9025a694","Type":"ContainerStarted","Data":"32686e7a4e0b6a5b7f1842ba42ba590095d1301376f1562ec3ffc50106914af8"}
Sep 17 23:13:33.745768 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.745701    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv" event={"ID":"32762502-7f12-4775-9465-1c8f9025a694","Type":"ContainerStarted","Data":"18b41b77dc90c6548c03b6dfa9481ef00cc03b11f548a0ae75702d93b723ca49"}
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757449    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757487    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757521    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757546    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757573    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757630    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757658    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757702    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757724    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757749    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757779    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757803    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757839    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757861    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7xk6\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-kube-api-access-w7xk6\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757899    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.758306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.759174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757961    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.759174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.757983    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.759174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.758007    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.759174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.758971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.760065 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.760036    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.760150 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.760067    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.762589 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.762569    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.762965 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.762939    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.764083 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.763550    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.764083 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.763962    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.764473 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.764449    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.765135 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.764855    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.765135 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.765096    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.767033 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.767009    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.767717 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.767673    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.768339 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.768161    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.768866 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.768821    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.769422 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.769367    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.769818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.769795    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.769964 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.769895    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.770421 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.770400    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7xk6\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-kube-api-access-w7xk6\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.770536 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.770520    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.771705 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.771657    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:33.815513 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.815468    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:33.875915 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:33.875816    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:34.176824 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.176800    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:13:34.179188 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:34.179162    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d43e04_1b0c_42e5_8cd0_f8607b3361b3.slice/crio-0b09bc6dadc2f86b38daf526c78e57f6e6c6f554307a76ad63a4c5d4abc1bdfd WatchSource:0}: Error finding container 0b09bc6dadc2f86b38daf526c78e57f6e6c6f554307a76ad63a4c5d4abc1bdfd: Status 404 returned error can't find the container with id 0b09bc6dadc2f86b38daf526c78e57f6e6c6f554307a76ad63a4c5d4abc1bdfd
Sep 17 23:13:34.757553 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.757517    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv" event={"ID":"32762502-7f12-4775-9465-1c8f9025a694","Type":"ContainerStarted","Data":"74ad4dbf454763656b1dbb2f7abcc4fffbc5a53651f6655467a7af6cea9f0218"}
Sep 17 23:13:34.757722 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.757629    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv" event={"ID":"32762502-7f12-4775-9465-1c8f9025a694","Type":"ContainerStarted","Data":"e6c4d76d190bd8ab216203ec4d09f7fa9a05db521530477c1f5ad9bcb4223120"}
Sep 17 23:13:34.757722 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.757681    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:34.759597 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.759567    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerStarted","Data":"0b09bc6dadc2f86b38daf526c78e57f6e6c6f554307a76ad63a4c5d4abc1bdfd"}
Sep 17 23:13:34.761264 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.761228    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn" event={"ID":"10d39d3d-7f7f-460e-b412-09c0280fdfbd","Type":"ContainerStarted","Data":"5f8a406790478501a6ae8b88bf22a609f8620a6979490bc557baafc0f55fdf1e"}
Sep 17 23:13:34.761556 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.761511    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:34.767171 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.767149    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn"
Sep 17 23:13:34.780952 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.780899    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv" podStartSLOduration=2.356227476 podStartE2EDuration="5.780883801s" totalImagesPullingTime="3.424656325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:34.779248254 +0000 UTC m=+139.240165987" watchObservedRunningTime="2026-09-17 23:13:34.780883801 +0000 UTC m=+139.241801533"
Sep 17 23:13:34.799276 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.799236    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-fspzn" podStartSLOduration=2.384647997 podStartE2EDuration="3.799222666s" totalImagesPullingTime="1.414574669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:34.798253958 +0000 UTC m=+139.259171690" watchObservedRunningTime="2026-09-17 23:13:34.799222666 +0000 UTC m=+139.260140400"
Sep 17 23:13:34.955950 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:34.955920    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:35.059684 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:35.059652    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:35.767718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:35.767682    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerID="4c48efc5128a83ef87b40c5043f3a6f3c80b0fe91ff7c923ff19687ee016fdc8" exitCode=0
Sep 17 23:13:35.768203 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:35.767779    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerDied","Data":"4c48efc5128a83ef87b40c5043f3a6f3c80b0fe91ff7c923ff19687ee016fdc8"}
Sep 17 23:13:36.348137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:36.348102    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:36.451031 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:36.450984    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:36.664089 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:36.664002    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-565bb49474-ht2vn"
Sep 17 23:13:36.879123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:36.879021    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:37.071377 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:37.071220    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:37.172534 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:37.172503    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:39.459462 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.459424    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd" podUID="e19a0639-aa9e-45d5-8c00-3679618e9abb" containerName="registry" containerID="cri-o://978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450" gracePeriod=30
Sep 17 23:13:39.692101 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.692079    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:13:39.790962 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.790880    2571 generic.go:356] "Generic (PLEG): container finished" podID="e19a0639-aa9e-45d5-8c00-3679618e9abb" containerID="978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450" exitCode=0
Sep 17 23:13:39.790962 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.790934    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd"
Sep 17 23:13:39.791163 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.790969    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd" event={"ID":"e19a0639-aa9e-45d5-8c00-3679618e9abb","Type":"ContainerDied","Data":"978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450"}
Sep 17 23:13:39.791163 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.791051    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7bff7b87f5-2qjbd" event={"ID":"e19a0639-aa9e-45d5-8c00-3679618e9abb","Type":"ContainerDied","Data":"b563bc3ed3f3ce520d165800a343013a38e4de3037988e8c309626940a6b4c61"}
Sep 17 23:13:39.791163 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.791078    2571 scope.go:117] "RemoveContainer" containerID="978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450"
Sep 17 23:13:39.794451 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.794427    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerStarted","Data":"b34a63066381f96d99fd018f949b5d5b755119a099bfeb39e8c00da516cd8fb2"}
Sep 17 23:13:39.794569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.794478    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerStarted","Data":"423508d892a5509d338b1f5b57ed2affdbcbedbd456a7fd4c44a61a44d4d09a9"}
Sep 17 23:13:39.794569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.794504    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerStarted","Data":"c0508d6b9793e9bb50770721dfb2f140d75b4ae38af3657dadbf76b56900c909"}
Sep 17 23:13:39.794569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.794527    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerStarted","Data":"6b9889a091246456fed383a4c6b47be28cca86adbbf4e46c0c82803548de05b2"}
Sep 17 23:13:39.794569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.794551    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerStarted","Data":"f12f910f79d9c383dc3fe3eb1e84e74cfa636130a4a5f8a599bb5912ea8605ab"}
Sep 17 23:13:39.794569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.794572    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerStarted","Data":"0e95baecf7eaec5ebce099e13b84e35d66edab22e060d3e8d3918b7e9cf28a13"}
Sep 17 23:13:39.799379 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.799362    2571 scope.go:117] "RemoveContainer" containerID="978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450"
Sep 17 23:13:39.799652 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:39.799627    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450\": container with ID starting with 978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450 not found: ID does not exist" containerID="978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450"
Sep 17 23:13:39.799718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.799661    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450"} err="failed to get container status \"978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450\": rpc error: code = NotFound desc = could not find container \"978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450\": container with ID starting with 978fdcb81095d004467331d6bc339b01f636e1262d20278408bba2a0349df450 not found: ID does not exist"
Sep 17 23:13:39.816494 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816470    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw6vq\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-kube-api-access-pw6vq\") pod \"e19a0639-aa9e-45d5-8c00-3679618e9abb\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") "
Sep 17 23:13:39.816606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816501    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-trusted-ca\") pod \"e19a0639-aa9e-45d5-8c00-3679618e9abb\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") "
Sep 17 23:13:39.816606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816530    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-installation-pull-secrets\") pod \"e19a0639-aa9e-45d5-8c00-3679618e9abb\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") "
Sep 17 23:13:39.816606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816552    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-bound-sa-token\") pod \"e19a0639-aa9e-45d5-8c00-3679618e9abb\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") "
Sep 17 23:13:39.816743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816693    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e19a0639-aa9e-45d5-8c00-3679618e9abb-ca-trust-extracted\") pod \"e19a0639-aa9e-45d5-8c00-3679618e9abb\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") "
Sep 17 23:13:39.816743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816730    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-image-registry-private-configuration\") pod \"e19a0639-aa9e-45d5-8c00-3679618e9abb\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") "
Sep 17 23:13:39.816840 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816792    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-certificates\") pod \"e19a0639-aa9e-45d5-8c00-3679618e9abb\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") "
Sep 17 23:13:39.816889 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816865    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") pod \"e19a0639-aa9e-45d5-8c00-3679618e9abb\" (UID: \"e19a0639-aa9e-45d5-8c00-3679618e9abb\") "
Sep 17 23:13:39.816975 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.816881    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e19a0639-aa9e-45d5-8c00-3679618e9abb" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:39.817186 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.817166    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-trusted-ca\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:13:39.817472 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.817448    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e19a0639-aa9e-45d5-8c00-3679618e9abb" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:13:39.819511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.819480    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e19a0639-aa9e-45d5-8c00-3679618e9abb" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:39.819606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.819560    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e19a0639-aa9e-45d5-8c00-3679618e9abb" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:39.819606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.819565    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e19a0639-aa9e-45d5-8c00-3679618e9abb" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:39.819606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.819574    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-kube-api-access-pw6vq" (OuterVolumeSpecName: "kube-api-access-pw6vq") pod "e19a0639-aa9e-45d5-8c00-3679618e9abb" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb"). InnerVolumeSpecName "kube-api-access-pw6vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:13:39.819606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.819578    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e19a0639-aa9e-45d5-8c00-3679618e9abb" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:13:39.825163 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.825140    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e19a0639-aa9e-45d5-8c00-3679618e9abb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e19a0639-aa9e-45d5-8c00-3679618e9abb" (UID: "e19a0639-aa9e-45d5-8c00-3679618e9abb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:13:39.830258 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.830220    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.405146783 podStartE2EDuration="6.830207586s" totalImagesPullingTime="4.407181063s" totalInitContainerRuntime="17.87974ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:39.826494406 +0000 UTC m=+144.287412136" watchObservedRunningTime="2026-09-17 23:13:39.830207586 +0000 UTC m=+144.291125359"
Sep 17 23:13:39.918942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.918829    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:13:39.918942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.918854    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw6vq\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-kube-api-access-pw6vq\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:13:39.918942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.918870    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-installation-pull-secrets\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:13:39.918942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.918885    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e19a0639-aa9e-45d5-8c00-3679618e9abb-bound-sa-token\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:13:39.918942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.918899    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e19a0639-aa9e-45d5-8c00-3679618e9abb-ca-trust-extracted\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:13:39.918942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.918912    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e19a0639-aa9e-45d5-8c00-3679618e9abb-image-registry-private-configuration\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:13:39.918942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:39.918929    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e19a0639-aa9e-45d5-8c00-3679618e9abb-registry-certificates\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:13:40.110522 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:40.110496    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7bff7b87f5-2qjbd"]
Sep 17 23:13:40.120782 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:40.120754    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7bff7b87f5-2qjbd"]
Sep 17 23:13:40.773821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:40.773794    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77f5b54f8c-vxsjv"
Sep 17 23:13:42.057651 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:42.057617    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e19a0639-aa9e-45d5-8c00-3679618e9abb" path="/var/lib/kubelet/pods/e19a0639-aa9e-45d5-8c00-3679618e9abb/volumes"
Sep 17 23:13:43.333297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:43.333254    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:43.412880 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:43.412851    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:43.876721 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:43.876695    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:13:47.572923 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:47.572880    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:47.700577 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:47.700523    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:51.373464 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:51.373415    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:51.480065 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:51.480016    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:52.214979 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:52.214943    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:52.372757 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:52.372720    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:53.384674 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:13:53.384642    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vv4bs" podUID="243d5166-07d6-4f35-8e7a-16b00a2961e9"
Sep 17 23:13:53.850316 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:53.850287    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:13:58.323565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:58.323529    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:58.359598 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:58.359574    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:13:58.362001 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:58.361969    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/243d5166-07d6-4f35-8e7a-16b00a2961e9-metrics-tls\") pod \"dns-default-vv4bs\" (UID: \"243d5166-07d6-4f35-8e7a-16b00a2961e9\") " pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:13:58.423493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:58.423459    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:58.653644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:58.653619    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9p4tt\""
Sep 17 23:13:58.661006 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:58.660989    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:13:58.939379 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:58.939315    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vv4bs"]
Sep 17 23:13:58.941252 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:13:58.941222    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243d5166_07d6_4f35_8e7a_16b00a2961e9.slice/crio-e8a48c5fbe66250e153390628ba738e8e1061936f176e2d692336b9c58cc40ab WatchSource:0}: Error finding container e8a48c5fbe66250e153390628ba738e8e1061936f176e2d692336b9c58cc40ab: Status 404 returned error can't find the container with id e8a48c5fbe66250e153390628ba738e8e1061936f176e2d692336b9c58cc40ab
Sep 17 23:13:59.356012 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:59.355972    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:59.431948 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:59.431924    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:13:59.876936 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:13:59.876898    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vv4bs" event={"ID":"243d5166-07d6-4f35-8e7a-16b00a2961e9","Type":"ContainerStarted","Data":"e8a48c5fbe66250e153390628ba738e8e1061936f176e2d692336b9c58cc40ab"}
Sep 17 23:14:00.883050 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:00.883023    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vv4bs" event={"ID":"243d5166-07d6-4f35-8e7a-16b00a2961e9","Type":"ContainerStarted","Data":"759c2c1b4f9e0262366af2ab7aea4e21b884fee67cb6b93ab9856f58acec4ea3"}
Sep 17 23:14:00.883550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:00.883072    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vv4bs" event={"ID":"243d5166-07d6-4f35-8e7a-16b00a2961e9","Type":"ContainerStarted","Data":"f1a3dff63ec27f471742abded8e170898cbcb0bf5377bdca36de0d031f138011"}
Sep 17 23:14:00.883550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:00.883132    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:14:00.901624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:00.901512    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vv4bs" podStartSLOduration=129.289942823 podStartE2EDuration="2m10.901493791s" totalImagesPullingTime="1.611550968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:00.900854839 +0000 UTC m=+165.361772574" watchObservedRunningTime="2026-09-17 23:14:00.901493791 +0000 UTC m=+165.362411524"
Sep 17 23:14:10.893924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:10.893832    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vv4bs"
Sep 17 23:14:11.476677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:11.476646    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:11.580534 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:11.580484    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:13.352506 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:13.352362    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:13.443022 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:13.442984    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:14.945181 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:14.945150    2571 generic.go:356] "Generic (PLEG): container finished" podID="74ed121c-9dd7-441c-b9a5-5e902465c1c0" containerID="0aace2f4535be90b0157bea5d33e7774a02ce281555b818ab0f2a3bf55762b05" exitCode=0
Sep 17 23:14:14.945579 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:14.945233    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj" event={"ID":"74ed121c-9dd7-441c-b9a5-5e902465c1c0","Type":"ContainerDied","Data":"0aace2f4535be90b0157bea5d33e7774a02ce281555b818ab0f2a3bf55762b05"}
Sep 17 23:14:14.945579 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:14.945546    2571 scope.go:117] "RemoveContainer" containerID="0aace2f4535be90b0157bea5d33e7774a02ce281555b818ab0f2a3bf55762b05"
Sep 17 23:14:15.951910 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:15.951874    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-q89mj" event={"ID":"74ed121c-9dd7-441c-b9a5-5e902465c1c0","Type":"ContainerStarted","Data":"607431edc7d2d9f9c0ff29363efa1ce0315a7693a825a28011473d2c7c6df341"}
Sep 17 23:14:20.555486 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:20.555451    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:33.877041 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:33.876999    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:33.897930 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:33.897897    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:34.037950 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:34.037925    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:43.407916 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:43.407870    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:43.511363 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:43.511308    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:50.942068 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:50.942030    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:51.042592 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.042560    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:51.142121 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.142096    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:51.694181 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.694148    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:51.794967 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.794940    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:51.870821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.870791    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:14:51.871097 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.871066    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="prometheus" containerID="cri-o://0e95baecf7eaec5ebce099e13b84e35d66edab22e060d3e8d3918b7e9cf28a13" gracePeriod=600
Sep 17 23:14:51.871228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.871083    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy" containerID="cri-o://423508d892a5509d338b1f5b57ed2affdbcbedbd456a7fd4c44a61a44d4d09a9" gracePeriod=600
Sep 17 23:14:51.871228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.871125    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="thanos-sidecar" containerID="cri-o://6b9889a091246456fed383a4c6b47be28cca86adbbf4e46c0c82803548de05b2" gracePeriod=600
Sep 17 23:14:51.871228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.871154    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b34a63066381f96d99fd018f949b5d5b755119a099bfeb39e8c00da516cd8fb2" gracePeriod=600
Sep 17 23:14:51.871228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.871208    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="config-reloader" containerID="cri-o://f12f910f79d9c383dc3fe3eb1e84e74cfa636130a4a5f8a599bb5912ea8605ab" gracePeriod=600
Sep 17 23:14:51.871463 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:51.871154    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy-web" containerID="cri-o://c0508d6b9793e9bb50770721dfb2f140d75b4ae38af3657dadbf76b56900c909" gracePeriod=600
Sep 17 23:14:52.100191 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100159    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerID="b34a63066381f96d99fd018f949b5d5b755119a099bfeb39e8c00da516cd8fb2" exitCode=0
Sep 17 23:14:52.100191 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100183    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerID="423508d892a5509d338b1f5b57ed2affdbcbedbd456a7fd4c44a61a44d4d09a9" exitCode=0
Sep 17 23:14:52.100191 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100189    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerID="c0508d6b9793e9bb50770721dfb2f140d75b4ae38af3657dadbf76b56900c909" exitCode=0
Sep 17 23:14:52.100191 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100196    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerID="6b9889a091246456fed383a4c6b47be28cca86adbbf4e46c0c82803548de05b2" exitCode=0
Sep 17 23:14:52.100191 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100201    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerID="f12f910f79d9c383dc3fe3eb1e84e74cfa636130a4a5f8a599bb5912ea8605ab" exitCode=0
Sep 17 23:14:52.100677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100207    2571 generic.go:356] "Generic (PLEG): container finished" podID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerID="0e95baecf7eaec5ebce099e13b84e35d66edab22e060d3e8d3918b7e9cf28a13" exitCode=0
Sep 17 23:14:52.100677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100232    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerDied","Data":"b34a63066381f96d99fd018f949b5d5b755119a099bfeb39e8c00da516cd8fb2"}
Sep 17 23:14:52.100677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100314    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerDied","Data":"423508d892a5509d338b1f5b57ed2affdbcbedbd456a7fd4c44a61a44d4d09a9"}
Sep 17 23:14:52.100677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100389    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerDied","Data":"c0508d6b9793e9bb50770721dfb2f140d75b4ae38af3657dadbf76b56900c909"}
Sep 17 23:14:52.100677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100430    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerDied","Data":"6b9889a091246456fed383a4c6b47be28cca86adbbf4e46c0c82803548de05b2"}
Sep 17 23:14:52.100677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100476    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerDied","Data":"f12f910f79d9c383dc3fe3eb1e84e74cfa636130a4a5f8a599bb5912ea8605ab"}
Sep 17 23:14:52.100677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.100514    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerDied","Data":"0e95baecf7eaec5ebce099e13b84e35d66edab22e060d3e8d3918b7e9cf28a13"}
Sep 17 23:14:52.122360 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.122340    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148464    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-kube-rbac-proxy\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148514    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-grpc-tls\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148548    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-serving-certs-ca-bundle\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148572    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148598    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-db\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148633    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-1\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148675    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-trusted-ca-bundle\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148704    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config-out\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148736    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-0\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148775    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148797    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-tls\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148822    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-kubelet-serving-ca-bundle\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148867    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-tls-assets\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148891    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148918    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-thanos-prometheus-http-client-file\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148954    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-metrics-client-certs\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.148991    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-web-config\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.150039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.149016    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-metrics-client-ca\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.151069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.149039    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7xk6\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-kube-api-access-w7xk6\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.151069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.149060    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-2\") pod \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\" (UID: \"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3\") "
Sep 17 23:14:52.151069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.149548    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:52.151069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.150624    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:52.151069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.150885    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:52.151444 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.151092    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:52.151444 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.151140    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:52.152707 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.152582    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:52.152842 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.152732    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:14:52.153467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.153438    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.153730 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.153703    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:14:52.155904 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.154911    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.155904 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.155199    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config" (OuterVolumeSpecName: "config") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.156489 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.156462    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-kube-api-access-w7xk6" (OuterVolumeSpecName: "kube-api-access-w7xk6") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "kube-api-access-w7xk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:52.156795 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.156747    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config-out" (OuterVolumeSpecName: "config-out") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:14:52.156795 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.156772    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.157496 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.157447    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.157649 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.157593    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:14:52.157649 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.157617    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.157811 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.157638    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.157986 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.157968    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.167137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.167106    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-web-config" (OuterVolumeSpecName: "web-config") pod "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" (UID: "d2d43e04-1b0c-42e5-8cd0-f8607b3361b3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:14:52.249974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249896    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.249974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249922    2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-kube-rbac-proxy\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.249974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249931    2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-grpc-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.249974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249940    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.249974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249950    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.249974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249960    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-db\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.249974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249968    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.249974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249976    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249985    2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config-out\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.249993    2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250000    2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250010    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250018    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250026    2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-tls-assets\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250034    2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250042    2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250050    2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-secret-metrics-client-certs\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250058    2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-web-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250066    2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-configmap-metrics-client-ca\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:52.250295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:52.250073    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7xk6\" (UniqueName: \"kubernetes.io/projected/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3-kube-api-access-w7xk6\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:14:53.107600 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.107574    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.108014 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.107576    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d2d43e04-1b0c-42e5-8cd0-f8607b3361b3","Type":"ContainerDied","Data":"0b09bc6dadc2f86b38daf526c78e57f6e6c6f554307a76ad63a4c5d4abc1bdfd"}
Sep 17 23:14:53.108014 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.107728    2571 scope.go:117] "RemoveContainer" containerID="b34a63066381f96d99fd018f949b5d5b755119a099bfeb39e8c00da516cd8fb2"
Sep 17 23:14:53.121468 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.121450    2571 scope.go:117] "RemoveContainer" containerID="423508d892a5509d338b1f5b57ed2affdbcbedbd456a7fd4c44a61a44d4d09a9"
Sep 17 23:14:53.128818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.128799    2571 scope.go:117] "RemoveContainer" containerID="c0508d6b9793e9bb50770721dfb2f140d75b4ae38af3657dadbf76b56900c909"
Sep 17 23:14:53.136115 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.136091    2571 scope.go:117] "RemoveContainer" containerID="6b9889a091246456fed383a4c6b47be28cca86adbbf4e46c0c82803548de05b2"
Sep 17 23:14:53.137692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.137635    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:14:53.143028 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.143006    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:14:53.143716 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.143703    2571 scope.go:117] "RemoveContainer" containerID="f12f910f79d9c383dc3fe3eb1e84e74cfa636130a4a5f8a599bb5912ea8605ab"
Sep 17 23:14:53.150467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.150453    2571 scope.go:117] "RemoveContainer" containerID="0e95baecf7eaec5ebce099e13b84e35d66edab22e060d3e8d3918b7e9cf28a13"
Sep 17 23:14:53.159688 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.159666    2571 scope.go:117] "RemoveContainer" containerID="4c48efc5128a83ef87b40c5043f3a6f3c80b0fe91ff7c923ff19687ee016fdc8"
Sep 17 23:14:53.172994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.172973    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:14:53.173256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173246    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19a0639-aa9e-45d5-8c00-3679618e9abb" containerName="registry"
Sep 17 23:14:53.173297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173259    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19a0639-aa9e-45d5-8c00-3679618e9abb" containerName="registry"
Sep 17 23:14:53.173297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173268    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="config-reloader"
Sep 17 23:14:53.173297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173273    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="config-reloader"
Sep 17 23:14:53.173297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173280    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="thanos-sidecar"
Sep 17 23:14:53.173297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173286    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="thanos-sidecar"
Sep 17 23:14:53.173297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173294    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy-web"
Sep 17 23:14:53.173297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173299    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy-web"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173308    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="prometheus"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173313    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="prometheus"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173338    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="init-config-reloader"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173345    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="init-config-reloader"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173350    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy-thanos"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173355    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy-thanos"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173362    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173367    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173409    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy-web"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173418    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="config-reloader"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173423    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="thanos-sidecar"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173429    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="prometheus"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173434    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy-thanos"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173439    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" containerName="kube-rbac-proxy"
Sep 17 23:14:53.173511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.173445    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19a0639-aa9e-45d5-8c00-3679618e9abb" containerName="registry"
Sep 17 23:14:53.178724 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.178704    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.183111 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183092    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 23:14:53.183193 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183092    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 23:14:53.183417 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183397    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 23:14:53.183515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183431    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 23:14:53.183515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183442    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 23:14:53.183515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183488    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 23:14:53.183515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183507    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 23:14:53.183694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183447    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 23:14:53.183694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183548    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 23:14:53.183694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183688    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 23:14:53.183844 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183712    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-db3mnal6kjvu5\""
Sep 17 23:14:53.183844 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183740    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-bsfvr\""
Sep 17 23:14:53.183844 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183746    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 23:14:53.183844 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183766    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 23:14:53.183844 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.183714    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 23:14:53.187392 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.187374    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 23:14:53.190814 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.190796    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 23:14:53.197559 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.197538    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:14:53.258517 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258494    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmtqq\" (UniqueName: \"kubernetes.io/projected/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-kube-api-access-rmtqq\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258609 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258522    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258609 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258540    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258609 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258583    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258649    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258691    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258754    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258780    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258795    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258808    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.258838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258827    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.259039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258840    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.259039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258861    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.259039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258896    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.259039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258912    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.259039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-config\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.259039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.258951    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.259039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.259001    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359686    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359713    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmtqq\" (UniqueName: \"kubernetes.io/projected/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-kube-api-access-rmtqq\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359731    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359747    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359765    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359782    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359911    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.359994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.359989    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360131 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360046    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360131 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360090    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360131 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360119    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360146    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360183    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360211    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360247    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360277    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360640 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360299    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360640 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360343    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-config\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360640 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360817 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360792    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.360878 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.360842    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.361554 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.361531    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.363713 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.363686    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.363969 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.363945    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.364086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.364066    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.364146 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.364096    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.364258 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.364232    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.364990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.364560    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.364990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.364936    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.365192 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.365042    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.366362 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.365462    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.366936 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.366708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.366936 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.366917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.367071 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.366917    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.367134 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.367117    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.367313 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.367283    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.367932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.367909    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-config\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.368287 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.368262    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.369163 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.369129    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmtqq\" (UniqueName: \"kubernetes.io/projected/c8ffd3bc-a89a-4f02-9b68-c4bac041d413-kube-api-access-rmtqq\") pod \"prometheus-k8s-0\" (UID: \"c8ffd3bc-a89a-4f02-9b68-c4bac041d413\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.490364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.490308    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:14:53.780094 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:53.780067    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 23:14:53.781375 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:14:53.781348    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ffd3bc_a89a_4f02_9b68_c4bac041d413.slice/crio-b12b960c5651af4af0da40abce2959e56732e57ff6b8969a425c130dcf2cdcad WatchSource:0}: Error finding container b12b960c5651af4af0da40abce2959e56732e57ff6b8969a425c130dcf2cdcad: Status 404 returned error can't find the container with id b12b960c5651af4af0da40abce2959e56732e57ff6b8969a425c130dcf2cdcad
Sep 17 23:14:54.057681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:54.057618    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d43e04-1b0c-42e5-8cd0-f8607b3361b3" path="/var/lib/kubelet/pods/d2d43e04-1b0c-42e5-8cd0-f8607b3361b3/volumes"
Sep 17 23:14:54.113938 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:54.113913    2571 generic.go:356] "Generic (PLEG): container finished" podID="c8ffd3bc-a89a-4f02-9b68-c4bac041d413" containerID="c6e200edd09a6751ab442d1211a726c5cd3bbd92bcffdcc9789d13337357c331" exitCode=0
Sep 17 23:14:54.114226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:54.113990    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8ffd3bc-a89a-4f02-9b68-c4bac041d413","Type":"ContainerDied","Data":"c6e200edd09a6751ab442d1211a726c5cd3bbd92bcffdcc9789d13337357c331"}
Sep 17 23:14:54.114226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:54.114045    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8ffd3bc-a89a-4f02-9b68-c4bac041d413","Type":"ContainerStarted","Data":"b12b960c5651af4af0da40abce2959e56732e57ff6b8969a425c130dcf2cdcad"}
Sep 17 23:14:55.121896 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.121860    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8ffd3bc-a89a-4f02-9b68-c4bac041d413","Type":"ContainerStarted","Data":"3cf5101f311eec6863191511dc1120184a63e8ae69fd919eb0225b63c9498a4e"}
Sep 17 23:14:55.122262 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.121922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8ffd3bc-a89a-4f02-9b68-c4bac041d413","Type":"ContainerStarted","Data":"4197c818424ab27c8e89347adda9f96d4446abcb3c9d40ea5c5705a568fdfe40"}
Sep 17 23:14:55.122262 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.121955    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8ffd3bc-a89a-4f02-9b68-c4bac041d413","Type":"ContainerStarted","Data":"377771cc78088d65c8062e9e8d97780d7e7b3453000d10e361e21daf3c001047"}
Sep 17 23:14:55.122262 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.121981    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8ffd3bc-a89a-4f02-9b68-c4bac041d413","Type":"ContainerStarted","Data":"110fedf2ad7511093d56ad383e55b28cf8ad8f56c633ef1e3d25e53d55cb3750"}
Sep 17 23:14:55.122262 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.122005    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8ffd3bc-a89a-4f02-9b68-c4bac041d413","Type":"ContainerStarted","Data":"f58a6cda1ef4df670460a805013d15fd08306f980b98293df27f4a6821995d7f"}
Sep 17 23:14:55.122262 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.122027    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8ffd3bc-a89a-4f02-9b68-c4bac041d413","Type":"ContainerStarted","Data":"77dba364d75ae9e1f6f0818186e937996e9ba225cb1861177ac44caa923de9e1"}
Sep 17 23:14:55.156707 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.156654    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.138108431 podStartE2EDuration="2.156636779s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.528348ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:55.153822397 +0000 UTC m=+219.614740129" watchObservedRunningTime="2026-09-17 23:14:55.156636779 +0000 UTC m=+219.617554509"
Sep 17 23:14:55.331660 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.331619    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:55.406197 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:55.406111    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:58.330473 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:58.330435    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:58.429444 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:58.429411    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:14:58.490779 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:14:58.490753    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:01.741962 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:15:01.741926    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:01.842812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:15:01.842781    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:02.020561 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:15:02.020486    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:53.491399 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:15:53.491283    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:53.508649 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:15:53.508619    2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:54.367448 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:15:54.367424    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 23:15:56.351304 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:15:56.351264    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:15:56.469041 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:15:56.469007    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:15.956926 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:15.956894    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:16:15.962481 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:15.962458    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:16:15.968627 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:15.968609    2571 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:16:23.662118 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:23.662075    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:23.901462 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:23.901417    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:31.326698 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:31.326643    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:31.605520 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:31.605480    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:31.718079 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:31.718043    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:32.071178 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:32.071090    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:32.173927 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:32.173902    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:32.787565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:32.787532    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:32.889904 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:32.889864    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:33.100692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:33.100655    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:33.296694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:33.296659    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:33.438723 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:33.438638    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:33.553968 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:33.553930    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:16:38.371956 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:16:38.371912    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:21.950463 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:21.950385    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:23.407015 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:23.406976    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.642849 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:45.642809    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.721220 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:45.721188    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.787064 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:45.787036    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.887860 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:45.887828    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:45.988599 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:45.988528    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:46.301670 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:46.301600    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:46.394175 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:46.394152    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:46.460393 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:46.460367    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:52.906514 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:52.906479    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:17:52.977005 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:17:52.976960    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:19.776679 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:18:19.776628    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:22.388172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:18:22.388133    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:22.496900 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:18:22.496863    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:45.962539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:18:45.962445    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:56.104952 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:18:56.104917    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:18:56.163614 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:18:56.163572    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:00.810963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:00.810913    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:00.883190 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:00.883160    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:00.939368 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:00.939340    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:01.123831 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:01.123791    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:01.243218 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:01.243187    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:01.935420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:01.935387    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:02.037021 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:02.036995    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:06.896182 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:06.896142    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:06.997900 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:06.997868    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:07.147469 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:07.147402    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:17.162208 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.162179    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-gs9w9"]
Sep 17 23:19:17.165364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.165315    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gs9w9"
Sep 17 23:19:17.167847 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.167823    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 23:19:17.167961 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.167946    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 23:19:17.168122 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.168105    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:19:17.169090 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.169076    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8hff\""
Sep 17 23:19:17.169157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.169134    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:19:17.173151 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.173131    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gs9w9"]
Sep 17 23:19:17.292802 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.292777    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7eec705-ebd5-4446-8a0a-0c77945132b8-model-cache\") pod \"s3-init-gs9w9\" (UID: \"c7eec705-ebd5-4446-8a0a-0c77945132b8\") " pod="kserve/s3-init-gs9w9"
Sep 17 23:19:17.292956 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.292834    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9vb\" (UniqueName: \"kubernetes.io/projected/c7eec705-ebd5-4446-8a0a-0c77945132b8-kube-api-access-bq9vb\") pod \"s3-init-gs9w9\" (UID: \"c7eec705-ebd5-4446-8a0a-0c77945132b8\") " pod="kserve/s3-init-gs9w9"
Sep 17 23:19:17.393377 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.393344    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7eec705-ebd5-4446-8a0a-0c77945132b8-model-cache\") pod \"s3-init-gs9w9\" (UID: \"c7eec705-ebd5-4446-8a0a-0c77945132b8\") " pod="kserve/s3-init-gs9w9"
Sep 17 23:19:17.393495 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.393410    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9vb\" (UniqueName: \"kubernetes.io/projected/c7eec705-ebd5-4446-8a0a-0c77945132b8-kube-api-access-bq9vb\") pod \"s3-init-gs9w9\" (UID: \"c7eec705-ebd5-4446-8a0a-0c77945132b8\") " pod="kserve/s3-init-gs9w9"
Sep 17 23:19:17.393640 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.393619    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7eec705-ebd5-4446-8a0a-0c77945132b8-model-cache\") pod \"s3-init-gs9w9\" (UID: \"c7eec705-ebd5-4446-8a0a-0c77945132b8\") " pod="kserve/s3-init-gs9w9"
Sep 17 23:19:17.403028 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.402997    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9vb\" (UniqueName: \"kubernetes.io/projected/c7eec705-ebd5-4446-8a0a-0c77945132b8-kube-api-access-bq9vb\") pod \"s3-init-gs9w9\" (UID: \"c7eec705-ebd5-4446-8a0a-0c77945132b8\") " pod="kserve/s3-init-gs9w9"
Sep 17 23:19:17.492640 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.492581    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gs9w9"
Sep 17 23:19:17.764300 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.764273    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-gs9w9"]
Sep 17 23:19:17.767038 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:19:17.766992    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7eec705_ebd5_4446_8a0a_0c77945132b8.slice/crio-07d50cb46437a1a2904d878d18ba5e02b07b147a32d4761f1d456055c121cf84 WatchSource:0}: Error finding container 07d50cb46437a1a2904d878d18ba5e02b07b147a32d4761f1d456055c121cf84: Status 404 returned error can't find the container with id 07d50cb46437a1a2904d878d18ba5e02b07b147a32d4761f1d456055c121cf84
Sep 17 23:19:17.769109 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:17.769095    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:19:18.159653 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:18.159622    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gs9w9" event={"ID":"c7eec705-ebd5-4446-8a0a-0c77945132b8","Type":"ContainerStarted","Data":"07d50cb46437a1a2904d878d18ba5e02b07b147a32d4761f1d456055c121cf84"}
Sep 17 23:19:20.681241 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:20.681201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:19:21.174498 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:21.174453    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gs9w9" event={"ID":"c7eec705-ebd5-4446-8a0a-0c77945132b8","Type":"ContainerStarted","Data":"435a18cf6fb092beef297e592333d368c907130b7c59faa386372b6671b5f867"}
Sep 17 23:19:23.184618 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:23.184590    2571 generic.go:356] "Generic (PLEG): container finished" podID="c7eec705-ebd5-4446-8a0a-0c77945132b8" containerID="435a18cf6fb092beef297e592333d368c907130b7c59faa386372b6671b5f867" exitCode=0
Sep 17 23:19:23.184928 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:23.184655    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gs9w9" event={"ID":"c7eec705-ebd5-4446-8a0a-0c77945132b8","Type":"ContainerDied","Data":"435a18cf6fb092beef297e592333d368c907130b7c59faa386372b6671b5f867"}
Sep 17 23:19:28.207256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:28.207223    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gs9w9" event={"ID":"c7eec705-ebd5-4446-8a0a-0c77945132b8","Type":"ContainerStarted","Data":"db316471263fa2280431599e768acfbc8492aafe0a9f2a1c0d876d1716b92265"}
Sep 17 23:19:28.227157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:19:28.227103    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-gs9w9" podStartSLOduration=1.838673062 podStartE2EDuration="11.22708496s" totalImagesPullingTime="7.685202002s" totalInitContainerRuntime="1.703209896s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:28.226228706 +0000 UTC m=+492.687146436" watchObservedRunningTime="2026-09-17 23:19:28.22708496 +0000 UTC m=+492.688002694"
Sep 17 23:20:33.053653 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:33.053601    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/s3-init-gs9w9" podUID="c7eec705-ebd5-4446-8a0a-0c77945132b8" containerName="s3-init" containerID="cri-o://db316471263fa2280431599e768acfbc8492aafe0a9f2a1c0d876d1716b92265" gracePeriod=30
Sep 17 23:20:33.464174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:33.464145    2571 generic.go:356] "Generic (PLEG): container finished" podID="c7eec705-ebd5-4446-8a0a-0c77945132b8" containerID="db316471263fa2280431599e768acfbc8492aafe0a9f2a1c0d876d1716b92265" exitCode=0
Sep 17 23:20:33.464307 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:33.464197    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gs9w9" event={"ID":"c7eec705-ebd5-4446-8a0a-0c77945132b8","Type":"ContainerDied","Data":"db316471263fa2280431599e768acfbc8492aafe0a9f2a1c0d876d1716b92265"}
Sep 17 23:20:33.464307 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:33.464228    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-gs9w9" event={"ID":"c7eec705-ebd5-4446-8a0a-0c77945132b8","Type":"ContainerDied","Data":"07d50cb46437a1a2904d878d18ba5e02b07b147a32d4761f1d456055c121cf84"}
Sep 17 23:20:33.464307 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:33.464240    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07d50cb46437a1a2904d878d18ba5e02b07b147a32d4761f1d456055c121cf84"
Sep 17 23:20:33.464307 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:33.464292    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gs9w9"
Sep 17 23:20:34.467555 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:34.467527    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gs9w9"
Sep 17 23:20:34.475305 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:34.475284    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gs9w9"
Sep 17 23:20:34.557440 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:34.557409    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq9vb\" (UniqueName: \"kubernetes.io/projected/c7eec705-ebd5-4446-8a0a-0c77945132b8-kube-api-access-bq9vb\") pod \"c7eec705-ebd5-4446-8a0a-0c77945132b8\" (UID: \"c7eec705-ebd5-4446-8a0a-0c77945132b8\") "
Sep 17 23:20:34.557543 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:34.557459    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7eec705-ebd5-4446-8a0a-0c77945132b8-model-cache\") pod \"c7eec705-ebd5-4446-8a0a-0c77945132b8\" (UID: \"c7eec705-ebd5-4446-8a0a-0c77945132b8\") "
Sep 17 23:20:34.559640 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:34.559616    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eec705-ebd5-4446-8a0a-0c77945132b8-kube-api-access-bq9vb" (OuterVolumeSpecName: "kube-api-access-bq9vb") pod "c7eec705-ebd5-4446-8a0a-0c77945132b8" (UID: "c7eec705-ebd5-4446-8a0a-0c77945132b8"). InnerVolumeSpecName "kube-api-access-bq9vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:20:34.576248 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:34.576223    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7eec705-ebd5-4446-8a0a-0c77945132b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7eec705-ebd5-4446-8a0a-0c77945132b8" (UID: "c7eec705-ebd5-4446-8a0a-0c77945132b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:34.658251 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:34.658229    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9vb\" (UniqueName: \"kubernetes.io/projected/c7eec705-ebd5-4446-8a0a-0c77945132b8-kube-api-access-bq9vb\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:20:34.658251 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:34.658250    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7eec705-ebd5-4446-8a0a-0c77945132b8-model-cache\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:20:35.471936 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:35.471901    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-gs9w9"
Sep 17 23:20:36.181883 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:20:36.181851    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:09.684009 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.683977    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-mhwkg"]
Sep 17 23:21:09.684507 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.684281    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7eec705-ebd5-4446-8a0a-0c77945132b8" containerName="download-hf-model"
Sep 17 23:21:09.684507 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.684292    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7eec705-ebd5-4446-8a0a-0c77945132b8" containerName="download-hf-model"
Sep 17 23:21:09.684507 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.684300    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7eec705-ebd5-4446-8a0a-0c77945132b8" containerName="s3-init"
Sep 17 23:21:09.684507 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.684306    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7eec705-ebd5-4446-8a0a-0c77945132b8" containerName="s3-init"
Sep 17 23:21:09.684507 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.684369    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7eec705-ebd5-4446-8a0a-0c77945132b8" containerName="s3-init"
Sep 17 23:21:09.687224 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.687207    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:09.689722 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.689700    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 23:21:09.691039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.691016    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:21:09.691039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.691042    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:21:09.691240 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.691042    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8hff\""
Sep 17 23:21:09.691240 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.691019    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 23:21:09.695358 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.695316    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-mhwkg"]
Sep 17 23:21:09.801560 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.801536    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxqbq\" (UniqueName: \"kubernetes.io/projected/7086040b-d590-4c64-b497-d8455053314e-kube-api-access-zxqbq\") pod \"s3-tls-init-custom-mhwkg\" (UID: \"7086040b-d590-4c64-b497-d8455053314e\") " pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:09.801665 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.801566    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7086040b-d590-4c64-b497-d8455053314e-model-cache\") pod \"s3-tls-init-custom-mhwkg\" (UID: \"7086040b-d590-4c64-b497-d8455053314e\") " pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:09.901867 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.901828    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxqbq\" (UniqueName: \"kubernetes.io/projected/7086040b-d590-4c64-b497-d8455053314e-kube-api-access-zxqbq\") pod \"s3-tls-init-custom-mhwkg\" (UID: \"7086040b-d590-4c64-b497-d8455053314e\") " pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:09.901969 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.901880    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7086040b-d590-4c64-b497-d8455053314e-model-cache\") pod \"s3-tls-init-custom-mhwkg\" (UID: \"7086040b-d590-4c64-b497-d8455053314e\") " pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:09.902272 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.902251    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7086040b-d590-4c64-b497-d8455053314e-model-cache\") pod \"s3-tls-init-custom-mhwkg\" (UID: \"7086040b-d590-4c64-b497-d8455053314e\") " pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:09.912815 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:09.912798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxqbq\" (UniqueName: \"kubernetes.io/projected/7086040b-d590-4c64-b497-d8455053314e-kube-api-access-zxqbq\") pod \"s3-tls-init-custom-mhwkg\" (UID: \"7086040b-d590-4c64-b497-d8455053314e\") " pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:10.009087 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:10.009036    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:10.281247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:10.281220    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-mhwkg"]
Sep 17 23:21:10.283123 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:21:10.283095    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7086040b_d590_4c64_b497_d8455053314e.slice/crio-5d4892ca05169f5ff67097847a1706ee072ae119ad0002e62858fbf996953956 WatchSource:0}: Error finding container 5d4892ca05169f5ff67097847a1706ee072ae119ad0002e62858fbf996953956: Status 404 returned error can't find the container with id 5d4892ca05169f5ff67097847a1706ee072ae119ad0002e62858fbf996953956
Sep 17 23:21:10.620109 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:10.620068    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mhwkg" event={"ID":"7086040b-d590-4c64-b497-d8455053314e","Type":"ContainerStarted","Data":"750682cfea8bf162c0df575b2a040ecbda7230894408aa8647e7cf1fb9d63f65"}
Sep 17 23:21:10.620266 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:10.620122    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mhwkg" event={"ID":"7086040b-d590-4c64-b497-d8455053314e","Type":"ContainerStarted","Data":"5d4892ca05169f5ff67097847a1706ee072ae119ad0002e62858fbf996953956"}
Sep 17 23:21:12.631283 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:12.631250    2571 generic.go:356] "Generic (PLEG): container finished" podID="7086040b-d590-4c64-b497-d8455053314e" containerID="750682cfea8bf162c0df575b2a040ecbda7230894408aa8647e7cf1fb9d63f65" exitCode=0
Sep 17 23:21:12.631704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:12.631348    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mhwkg" event={"ID":"7086040b-d590-4c64-b497-d8455053314e","Type":"ContainerDied","Data":"750682cfea8bf162c0df575b2a040ecbda7230894408aa8647e7cf1fb9d63f65"}
Sep 17 23:21:13.637529 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:13.637498    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mhwkg" event={"ID":"7086040b-d590-4c64-b497-d8455053314e","Type":"ContainerStarted","Data":"15a22b3a5af9319ba825661cd371e98bcf27ef248750bb83f68c0f63429b1539"}
Sep 17 23:21:13.655739 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:13.655696    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-mhwkg" podStartSLOduration=3.057346165 podStartE2EDuration="4.655681364s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.598335199s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:13.65416389 +0000 UTC m=+598.115081621" watchObservedRunningTime="2026-09-17 23:21:13.655681364 +0000 UTC m=+598.116599095"
Sep 17 23:21:15.986208 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:15.986181    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:21:15.989630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:15.989609    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:21:17.164698 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:17.164665    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:17.290397 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:17.290361    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:18.660635 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:18.660607    2571 generic.go:356] "Generic (PLEG): container finished" podID="7086040b-d590-4c64-b497-d8455053314e" containerID="15a22b3a5af9319ba825661cd371e98bcf27ef248750bb83f68c0f63429b1539" exitCode=0
Sep 17 23:21:18.660955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:18.660677    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mhwkg" event={"ID":"7086040b-d590-4c64-b497-d8455053314e","Type":"ContainerDied","Data":"15a22b3a5af9319ba825661cd371e98bcf27ef248750bb83f68c0f63429b1539"}
Sep 17 23:21:19.803210 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:19.803188    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:19.879894 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:19.879863    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxqbq\" (UniqueName: \"kubernetes.io/projected/7086040b-d590-4c64-b497-d8455053314e-kube-api-access-zxqbq\") pod \"7086040b-d590-4c64-b497-d8455053314e\" (UID: \"7086040b-d590-4c64-b497-d8455053314e\") "
Sep 17 23:21:19.880025 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:19.879924    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7086040b-d590-4c64-b497-d8455053314e-model-cache\") pod \"7086040b-d590-4c64-b497-d8455053314e\" (UID: \"7086040b-d590-4c64-b497-d8455053314e\") "
Sep 17 23:21:19.882028 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:19.881999    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7086040b-d590-4c64-b497-d8455053314e-kube-api-access-zxqbq" (OuterVolumeSpecName: "kube-api-access-zxqbq") pod "7086040b-d590-4c64-b497-d8455053314e" (UID: "7086040b-d590-4c64-b497-d8455053314e"). InnerVolumeSpecName "kube-api-access-zxqbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:19.902302 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:19.902274    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7086040b-d590-4c64-b497-d8455053314e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7086040b-d590-4c64-b497-d8455053314e" (UID: "7086040b-d590-4c64-b497-d8455053314e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:21:19.981302 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:19.981250    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxqbq\" (UniqueName: \"kubernetes.io/projected/7086040b-d590-4c64-b497-d8455053314e-kube-api-access-zxqbq\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:21:19.981302 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:19.981272    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7086040b-d590-4c64-b497-d8455053314e-model-cache\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:21:20.672409 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:20.672384    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-mhwkg"
Sep 17 23:21:20.672581 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:20.672403    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-mhwkg" event={"ID":"7086040b-d590-4c64-b497-d8455053314e","Type":"ContainerDied","Data":"5d4892ca05169f5ff67097847a1706ee072ae119ad0002e62858fbf996953956"}
Sep 17 23:21:20.672581 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:20.672451    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d4892ca05169f5ff67097847a1706ee072ae119ad0002e62858fbf996953956"
Sep 17 23:21:21.263040 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:21.263009    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:21.532206 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:21.532127    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:23.941440 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.941406    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-84xrw"]
Sep 17 23:21:23.943758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.941725    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7086040b-d590-4c64-b497-d8455053314e" containerName="s3-tls-init-custom"
Sep 17 23:21:23.943758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.941738    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7086040b-d590-4c64-b497-d8455053314e" containerName="s3-tls-init-custom"
Sep 17 23:21:23.943758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.941754    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7086040b-d590-4c64-b497-d8455053314e" containerName="download-hf-model"
Sep 17 23:21:23.943758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.941759    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7086040b-d590-4c64-b497-d8455053314e" containerName="download-hf-model"
Sep 17 23:21:23.943758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.941809    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7086040b-d590-4c64-b497-d8455053314e" containerName="s3-tls-init-custom"
Sep 17 23:21:23.944717 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.944689    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:23.947414 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.947393    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 23:21:23.947546 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.947400    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:21:23.947718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.947700    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 23:21:23.948697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.948677    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:21:23.948786 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.948748    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8hff\""
Sep 17 23:21:23.952671 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:23.952645    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-84xrw"]
Sep 17 23:21:24.109048 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.109016    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e31921b-bb2e-4769-9649-fb77c61c1e68-model-cache\") pod \"s3-tls-init-serving-84xrw\" (UID: \"2e31921b-bb2e-4769-9649-fb77c61c1e68\") " pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:24.109218 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.109053    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxr8\" (UniqueName: \"kubernetes.io/projected/2e31921b-bb2e-4769-9649-fb77c61c1e68-kube-api-access-4fxr8\") pod \"s3-tls-init-serving-84xrw\" (UID: \"2e31921b-bb2e-4769-9649-fb77c61c1e68\") " pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:24.209458 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.209395    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e31921b-bb2e-4769-9649-fb77c61c1e68-model-cache\") pod \"s3-tls-init-serving-84xrw\" (UID: \"2e31921b-bb2e-4769-9649-fb77c61c1e68\") " pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:24.209458 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.209428    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxr8\" (UniqueName: \"kubernetes.io/projected/2e31921b-bb2e-4769-9649-fb77c61c1e68-kube-api-access-4fxr8\") pod \"s3-tls-init-serving-84xrw\" (UID: \"2e31921b-bb2e-4769-9649-fb77c61c1e68\") " pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:24.209773 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.209754    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e31921b-bb2e-4769-9649-fb77c61c1e68-model-cache\") pod \"s3-tls-init-serving-84xrw\" (UID: \"2e31921b-bb2e-4769-9649-fb77c61c1e68\") " pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:24.219262 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.219240    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxr8\" (UniqueName: \"kubernetes.io/projected/2e31921b-bb2e-4769-9649-fb77c61c1e68-kube-api-access-4fxr8\") pod \"s3-tls-init-serving-84xrw\" (UID: \"2e31921b-bb2e-4769-9649-fb77c61c1e68\") " pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:24.264164 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.264132    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:24.536674 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.536653    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-84xrw"]
Sep 17 23:21:24.538701 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:21:24.538669    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e31921b_bb2e_4769_9649_fb77c61c1e68.slice/crio-6a20369f3aa22a44dad03d0c253340219766b039f91fcdc862b0ed03eff3d014 WatchSource:0}: Error finding container 6a20369f3aa22a44dad03d0c253340219766b039f91fcdc862b0ed03eff3d014: Status 404 returned error can't find the container with id 6a20369f3aa22a44dad03d0c253340219766b039f91fcdc862b0ed03eff3d014
Sep 17 23:21:24.691275 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.691236    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-84xrw" event={"ID":"2e31921b-bb2e-4769-9649-fb77c61c1e68","Type":"ContainerStarted","Data":"2d1d92bd7bef7608d7d193c1409ac9408ee46bf1257fbb4f173518433f31ca36"}
Sep 17 23:21:24.691397 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:24.691288    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-84xrw" event={"ID":"2e31921b-bb2e-4769-9649-fb77c61c1e68","Type":"ContainerStarted","Data":"6a20369f3aa22a44dad03d0c253340219766b039f91fcdc862b0ed03eff3d014"}
Sep 17 23:21:26.701150 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:26.701118    2571 generic.go:356] "Generic (PLEG): container finished" podID="2e31921b-bb2e-4769-9649-fb77c61c1e68" containerID="2d1d92bd7bef7608d7d193c1409ac9408ee46bf1257fbb4f173518433f31ca36" exitCode=0
Sep 17 23:21:26.701542 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:26.701202    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-84xrw" event={"ID":"2e31921b-bb2e-4769-9649-fb77c61c1e68","Type":"ContainerDied","Data":"2d1d92bd7bef7608d7d193c1409ac9408ee46bf1257fbb4f173518433f31ca36"}
Sep 17 23:21:27.707543 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:27.707510    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-84xrw" event={"ID":"2e31921b-bb2e-4769-9649-fb77c61c1e68","Type":"ContainerStarted","Data":"05c039144e898d2de13087f9302ec8ca056100c7ce797c1fbb01cc293c8011ad"}
Sep 17 23:21:27.725672 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:27.725623    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-84xrw" podStartSLOduration=3.04688361 podStartE2EDuration="4.725607448s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.678723838s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:27.724182635 +0000 UTC m=+612.185100366" watchObservedRunningTime="2026-09-17 23:21:27.725607448 +0000 UTC m=+612.186525179"
Sep 17 23:21:33.734460 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:33.734426    2571 generic.go:356] "Generic (PLEG): container finished" podID="2e31921b-bb2e-4769-9649-fb77c61c1e68" containerID="05c039144e898d2de13087f9302ec8ca056100c7ce797c1fbb01cc293c8011ad" exitCode=0
Sep 17 23:21:33.734867 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:33.734497    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-84xrw" event={"ID":"2e31921b-bb2e-4769-9649-fb77c61c1e68","Type":"ContainerDied","Data":"05c039144e898d2de13087f9302ec8ca056100c7ce797c1fbb01cc293c8011ad"}
Sep 17 23:21:34.866228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:34.866205    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:34.894184 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:34.894161    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fxr8\" (UniqueName: \"kubernetes.io/projected/2e31921b-bb2e-4769-9649-fb77c61c1e68-kube-api-access-4fxr8\") pod \"2e31921b-bb2e-4769-9649-fb77c61c1e68\" (UID: \"2e31921b-bb2e-4769-9649-fb77c61c1e68\") "
Sep 17 23:21:34.894286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:34.894252    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e31921b-bb2e-4769-9649-fb77c61c1e68-model-cache\") pod \"2e31921b-bb2e-4769-9649-fb77c61c1e68\" (UID: \"2e31921b-bb2e-4769-9649-fb77c61c1e68\") "
Sep 17 23:21:34.896218 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:34.896191    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e31921b-bb2e-4769-9649-fb77c61c1e68-kube-api-access-4fxr8" (OuterVolumeSpecName: "kube-api-access-4fxr8") pod "2e31921b-bb2e-4769-9649-fb77c61c1e68" (UID: "2e31921b-bb2e-4769-9649-fb77c61c1e68"). InnerVolumeSpecName "kube-api-access-4fxr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:34.916436 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:34.916412    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e31921b-bb2e-4769-9649-fb77c61c1e68-model-cache" (OuterVolumeSpecName: "model-cache") pod "2e31921b-bb2e-4769-9649-fb77c61c1e68" (UID: "2e31921b-bb2e-4769-9649-fb77c61c1e68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:21:34.995265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:34.995220    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxr8\" (UniqueName: \"kubernetes.io/projected/2e31921b-bb2e-4769-9649-fb77c61c1e68-kube-api-access-4fxr8\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:21:34.995265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:34.995239    2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e31921b-bb2e-4769-9649-fb77c61c1e68-model-cache\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:21:35.745099 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:35.745063    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-84xrw" event={"ID":"2e31921b-bb2e-4769-9649-fb77c61c1e68","Type":"ContainerDied","Data":"6a20369f3aa22a44dad03d0c253340219766b039f91fcdc862b0ed03eff3d014"}
Sep 17 23:21:35.745099 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:35.745079    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-84xrw"
Sep 17 23:21:35.745099 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:35.745105    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a20369f3aa22a44dad03d0c253340219766b039f91fcdc862b0ed03eff3d014"
Sep 17 23:21:36.088753 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:36.088719    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:36.190541 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:36.190511    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:47.425554 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.425454    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:47.517597 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.517570    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"]
Sep 17 23:21:47.517858 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.517847    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e31921b-bb2e-4769-9649-fb77c61c1e68" containerName="s3-tls-init-serving"
Sep 17 23:21:47.517902 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.517860    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e31921b-bb2e-4769-9649-fb77c61c1e68" containerName="s3-tls-init-serving"
Sep 17 23:21:47.517902 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.517874    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e31921b-bb2e-4769-9649-fb77c61c1e68" containerName="download-hf-model"
Sep 17 23:21:47.517902 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.517880    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e31921b-bb2e-4769-9649-fb77c61c1e68" containerName="download-hf-model"
Sep 17 23:21:47.517996 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.517921    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e31921b-bb2e-4769-9649-fb77c61c1e68" containerName="s3-tls-init-serving"
Sep 17 23:21:47.521250 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.521236    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.524020 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.523996    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:21:47.524132 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.524102    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:21:47.524360 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.524343    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 23:21:47.524424 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.524405    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:21:47.525188 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.525169    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:21:47.525240 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.525186    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:21:47.525240 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.525208    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 23:21:47.532603 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.532582    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"]
Sep 17 23:21:47.580853 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.580833    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfmcr\" (UniqueName: \"kubernetes.io/projected/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-api-access-hfmcr\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.580952 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.580864    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.580952 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.580894    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.581027 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.580973    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b44bc1-044c-4a0e-810d-84ce8da88d75-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.581027 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.581005    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.624860 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.624837    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:47.681560 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.681500    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.681658 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.681588    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b44bc1-044c-4a0e-810d-84ce8da88d75-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.681658 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.681618    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.681658 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.681645    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfmcr\" (UniqueName: \"kubernetes.io/projected/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-api-access-hfmcr\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.681813 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.681683    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.682064 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.682044    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.682383 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.682270    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.682383 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.682281    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.684208 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.684189    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b44bc1-044c-4a0e-810d-84ce8da88d75-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.691335 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.691296    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfmcr\" (UniqueName: \"kubernetes.io/projected/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-api-access-hfmcr\") pod \"isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.695240 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.695212    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"]
Sep 17 23:21:47.698954 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.698928    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.699074 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.698992    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:47.701434 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.701416    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-d56fb3-dockercfg-qbqmq\""
Sep 17 23:21:47.701624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.701605    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-d56fb3\""
Sep 17 23:21:47.701692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.701675    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d56fb3-kube-rbac-proxy-sar-config\""
Sep 17 23:21:47.701743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.701727    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-d56fb3-predictor-serving-cert\""
Sep 17 23:21:47.709187 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.709169    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"]
Sep 17 23:21:47.763365 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.763337    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:21:47.782511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.782492    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-cabundle-cert\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.782595 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.782518    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3912b145-ce1e-476f-b935-f5f3b84c9de4-kserve-provision-location\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.782595 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.782570    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3912b145-ce1e-476f-b935-f5f3b84c9de4-proxy-tls\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.782685 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.782611    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.782685 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.782628    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x66g4\" (UniqueName: \"kubernetes.io/projected/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-api-access-x66g4\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.831009 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.830985    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:21:47.883292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.883261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3912b145-ce1e-476f-b935-f5f3b84c9de4-proxy-tls\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.883445 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.883349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.883445 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.883386    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x66g4\" (UniqueName: \"kubernetes.io/projected/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-api-access-x66g4\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.883445 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.883427    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-cabundle-cert\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.883610 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.883454    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3912b145-ce1e-476f-b935-f5f3b84c9de4-kserve-provision-location\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.884056 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.883874    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3912b145-ce1e-476f-b935-f5f3b84c9de4-kserve-provision-location\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.884754 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.884712    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.885147 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.885124    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-cabundle-cert\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.886630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.886606    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3912b145-ce1e-476f-b935-f5f3b84c9de4-proxy-tls\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:47.897383 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:47.897365    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x66g4\" (UniqueName: \"kubernetes.io/projected/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-api-access-x66g4\") pod \"isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") " pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:48.009820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:48.009767    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:21:48.110389 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:48.110341    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"]
Sep 17 23:21:48.112628 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:21:48.112605    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b44bc1_044c_4a0e_810d_84ce8da88d75.slice/crio-1d6a7242a49ed4dace36a58e77b16b941ab10d7a7cffc2c7674b6c3cdbbf636b WatchSource:0}: Error finding container 1d6a7242a49ed4dace36a58e77b16b941ab10d7a7cffc2c7674b6c3cdbbf636b: Status 404 returned error can't find the container with id 1d6a7242a49ed4dace36a58e77b16b941ab10d7a7cffc2c7674b6c3cdbbf636b
Sep 17 23:21:48.289467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:48.289390    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"]
Sep 17 23:21:48.292007 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:21:48.291979    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3912b145_ce1e_476f_b935_f5f3b84c9de4.slice/crio-9db3c3c228bf52758e0de7954ca06caf4c53b2991568c38610729191bc7001d5 WatchSource:0}: Error finding container 9db3c3c228bf52758e0de7954ca06caf4c53b2991568c38610729191bc7001d5: Status 404 returned error can't find the container with id 9db3c3c228bf52758e0de7954ca06caf4c53b2991568c38610729191bc7001d5
Sep 17 23:21:48.806600 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:48.806563    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerStarted","Data":"366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606"}
Sep 17 23:21:48.807078 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:48.806646    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerStarted","Data":"1d6a7242a49ed4dace36a58e77b16b941ab10d7a7cffc2c7674b6c3cdbbf636b"}
Sep 17 23:21:48.808155 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:48.808131    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb" event={"ID":"3912b145-ce1e-476f-b935-f5f3b84c9de4","Type":"ContainerStarted","Data":"22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92"}
Sep 17 23:21:48.808249 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:48.808186    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb" event={"ID":"3912b145-ce1e-476f-b935-f5f3b84c9de4","Type":"ContainerStarted","Data":"9db3c3c228bf52758e0de7954ca06caf4c53b2991568c38610729191bc7001d5"}
Sep 17 23:21:52.827067 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:52.827031    2571 generic.go:356] "Generic (PLEG): container finished" podID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerID="366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606" exitCode=0
Sep 17 23:21:52.827527 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:52.827109    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerDied","Data":"366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606"}
Sep 17 23:21:53.840523 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:53.840495    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb_3912b145-ce1e-476f-b935-f5f3b84c9de4/storage-initializer/0.log"
Sep 17 23:21:53.840967 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:53.840541    2571 generic.go:356] "Generic (PLEG): container finished" podID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerID="22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92" exitCode=1
Sep 17 23:21:53.840967 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:53.840602    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb" event={"ID":"3912b145-ce1e-476f-b935-f5f3b84c9de4","Type":"ContainerDied","Data":"22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92"}
Sep 17 23:21:54.852278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:54.852247    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb_3912b145-ce1e-476f-b935-f5f3b84c9de4/storage-initializer/0.log"
Sep 17 23:21:54.852815 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:54.852361    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb" event={"ID":"3912b145-ce1e-476f-b935-f5f3b84c9de4","Type":"ContainerStarted","Data":"771dfdcb2eb67f6ac54167163163bd6245708208563592b01711718379ef5ea2"}
Sep 17 23:21:57.250215 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:57.250178    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"]
Sep 17 23:21:57.250700 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:57.250439    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb" podUID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerName="storage-initializer" containerID="cri-o://771dfdcb2eb67f6ac54167163163bd6245708208563592b01711718379ef5ea2" gracePeriod=30
Sep 17 23:21:57.290674 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:21:57.290628    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:01.890832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:01.890799    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb_3912b145-ce1e-476f-b935-f5f3b84c9de4/storage-initializer/1.log"
Sep 17 23:22:01.891360 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:01.891285    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb_3912b145-ce1e-476f-b935-f5f3b84c9de4/storage-initializer/0.log"
Sep 17 23:22:01.891360 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:01.891347    2571 generic.go:356] "Generic (PLEG): container finished" podID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerID="771dfdcb2eb67f6ac54167163163bd6245708208563592b01711718379ef5ea2" exitCode=1
Sep 17 23:22:01.891503 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:01.891460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb" event={"ID":"3912b145-ce1e-476f-b935-f5f3b84c9de4","Type":"ContainerDied","Data":"771dfdcb2eb67f6ac54167163163bd6245708208563592b01711718379ef5ea2"}
Sep 17 23:22:01.891597 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:01.891582    2571 scope.go:117] "RemoveContainer" containerID="22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92"
Sep 17 23:22:02.559720 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.559687    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:02.874357 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.874316    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb_3912b145-ce1e-476f-b935-f5f3b84c9de4/storage-initializer/1.log"
Sep 17 23:22:02.878517 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.878492    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:22:02.892204 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.892168    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:02.897543 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.897524    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb_3912b145-ce1e-476f-b935-f5f3b84c9de4/storage-initializer/1.log"
Sep 17 23:22:02.898027 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.897958    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb" event={"ID":"3912b145-ce1e-476f-b935-f5f3b84c9de4","Type":"ContainerDied","Data":"9db3c3c228bf52758e0de7954ca06caf4c53b2991568c38610729191bc7001d5"}
Sep 17 23:22:02.898133 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.898028    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"
Sep 17 23:22:02.909830 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.909808    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-cabundle-cert\") pod \"3912b145-ce1e-476f-b935-f5f3b84c9de4\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") "
Sep 17 23:22:02.909955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.909842    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-rbac-proxy-sar-config\") pod \"3912b145-ce1e-476f-b935-f5f3b84c9de4\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") "
Sep 17 23:22:02.909955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.909886    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3912b145-ce1e-476f-b935-f5f3b84c9de4-kserve-provision-location\") pod \"3912b145-ce1e-476f-b935-f5f3b84c9de4\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") "
Sep 17 23:22:02.910082 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.910005    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3912b145-ce1e-476f-b935-f5f3b84c9de4-proxy-tls\") pod \"3912b145-ce1e-476f-b935-f5f3b84c9de4\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") "
Sep 17 23:22:02.910082 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.910070    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x66g4\" (UniqueName: \"kubernetes.io/projected/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-api-access-x66g4\") pod \"3912b145-ce1e-476f-b935-f5f3b84c9de4\" (UID: \"3912b145-ce1e-476f-b935-f5f3b84c9de4\") "
Sep 17 23:22:02.910199 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.910132    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3912b145-ce1e-476f-b935-f5f3b84c9de4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3912b145-ce1e-476f-b935-f5f3b84c9de4" (UID: "3912b145-ce1e-476f-b935-f5f3b84c9de4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:22:02.910256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.910242    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3912b145-ce1e-476f-b935-f5f3b84c9de4" (UID: "3912b145-ce1e-476f-b935-f5f3b84c9de4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:22:02.910314 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.910254    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3912b145-ce1e-476f-b935-f5f3b84c9de4" (UID: "3912b145-ce1e-476f-b935-f5f3b84c9de4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:22:02.910395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.910365    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:22:02.910395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.910381    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:22:02.910505 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.910399    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3912b145-ce1e-476f-b935-f5f3b84c9de4-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:22:02.912990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.912958    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3912b145-ce1e-476f-b935-f5f3b84c9de4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3912b145-ce1e-476f-b935-f5f3b84c9de4" (UID: "3912b145-ce1e-476f-b935-f5f3b84c9de4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:22:02.913639 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:02.913609    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-api-access-x66g4" (OuterVolumeSpecName: "kube-api-access-x66g4") pod "3912b145-ce1e-476f-b935-f5f3b84c9de4" (UID: "3912b145-ce1e-476f-b935-f5f3b84c9de4"). InnerVolumeSpecName "kube-api-access-x66g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:22:03.011269 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:03.011233    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3912b145-ce1e-476f-b935-f5f3b84c9de4-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:22:03.011269 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:03.011265    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x66g4\" (UniqueName: \"kubernetes.io/projected/3912b145-ce1e-476f-b935-f5f3b84c9de4-kube-api-access-x66g4\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:22:03.243514 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:03.243483    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"]
Sep 17 23:22:03.245284 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:03.245260    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-d56fb3-predictor-84884df6d7-v4zhb"]
Sep 17 23:22:04.059394 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:04.059361    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3912b145-ce1e-476f-b935-f5f3b84c9de4" path="/var/lib/kubelet/pods/3912b145-ce1e-476f-b935-f5f3b84c9de4/volumes"
Sep 17 23:22:05.886051 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:05.885876    2571 scope.go:117] "RemoveContainer" containerID="771dfdcb2eb67f6ac54167163163bd6245708208563592b01711718379ef5ea2"
Sep 17 23:22:05.894222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:05.894201    2571 scope.go:117] "RemoveContainer" containerID="22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92"
Sep 17 23:22:05.894524 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:22:05.894504    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92\": container with ID starting with 22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92 not found: ID does not exist" containerID="22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92"
Sep 17 23:22:05.894584 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:05.894542    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92"} err="failed to get container status \"22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92\": rpc error: code = NotFound desc = could not find container \"22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92\": container with ID starting with 22f703929b71339b9ffa6fce0420047af81e9ab61a1f87ea9204dfdfd3df4e92 not found: ID does not exist"
Sep 17 23:22:06.919903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:06.919858    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerStarted","Data":"588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943"}
Sep 17 23:22:08.931409 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:08.931334    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerStarted","Data":"eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f"}
Sep 17 23:22:11.946837 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:11.946802    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerStarted","Data":"f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab"}
Sep 17 23:22:11.947247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:11.947081    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:22:11.947247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:11.947146    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:22:11.947247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:11.947183    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:22:11.948718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:11.948683    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:22:11.949394 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:11.949369    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:22:11.968642 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:11.968587    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podStartSLOduration=2.766400544 podStartE2EDuration="24.968573095s" totalImagesPullingTime="18.159470006s" totalInitContainerRuntime="4.042702545s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:22:11.966612065 +0000 UTC m=+656.427529792" watchObservedRunningTime="2026-09-17 23:22:11.968573095 +0000 UTC m=+656.429490831"
Sep 17 23:22:12.951288 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:12.951235    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:22:12.951810 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:12.951784    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:22:13.392546 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:13.392512    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:22:17.955632 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:17.955602    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:22:17.956102 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:17.956032    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:22:17.956367 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:17.956315    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:22:27.956743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:27.956691    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:22:27.957220 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:27.957190    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:22:37.956721 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:37.956669    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:22:37.957376 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:37.957085    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:22:47.956226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:47.956180    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:22:47.956689 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:47.956457    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:22:57.956392 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:57.956356    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:22:57.956816 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:22:57.956792    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:07.956909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:07.956835    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:23:07.957387 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:07.957364    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:17.956556 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:17.956522    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:23:17.957079 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:17.957021    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:23:28.728159 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:28.728120    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:30.033822 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:30.033788    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:32.411609 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.411580    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"]
Sep 17 23:23:32.411990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.411858    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" containerID="cri-o://f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab" gracePeriod=30
Sep 17 23:23:32.411990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.411888    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy" containerID="cri-o://eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f" gracePeriod=30
Sep 17 23:23:32.411990 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.411843    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" containerID="cri-o://588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943" gracePeriod=30
Sep 17 23:23:32.422773 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.422750    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:32.535017 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.534993    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:32.584089 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.584066    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"]
Sep 17 23:23:32.584435 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.584411    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerName="storage-initializer"
Sep 17 23:23:32.584435 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.584437    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerName="storage-initializer"
Sep 17 23:23:32.584529 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.584468    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerName="storage-initializer"
Sep 17 23:23:32.584529 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.584474    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerName="storage-initializer"
Sep 17 23:23:32.584529 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.584525    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerName="storage-initializer"
Sep 17 23:23:32.584631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.584539    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3912b145-ce1e-476f-b935-f5f3b84c9de4" containerName="storage-initializer"
Sep 17 23:23:32.587675 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.587661    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.590054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.590036    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 23:23:32.590157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.590060    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 23:23:32.605034 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.605015    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"]
Sep 17 23:23:32.654536 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.654508    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.654631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.654574    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hj86\" (UniqueName: \"kubernetes.io/projected/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-api-access-5hj86\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.654631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.654619    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe9636f-eab1-4c19-a74b-8f24b051449f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.654721 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.654638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.654721 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.654661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe9636f-eab1-4c19-a74b-8f24b051449f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.755788 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.755730    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hj86\" (UniqueName: \"kubernetes.io/projected/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-api-access-5hj86\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.755788 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.755767    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe9636f-eab1-4c19-a74b-8f24b051449f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.755788 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.755785    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.755993 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.755806    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe9636f-eab1-4c19-a74b-8f24b051449f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.755993 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.755836    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.756232 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.756209    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe9636f-eab1-4c19-a74b-8f24b051449f-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.756465 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.756446    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.756545 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.756528    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.758337 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.758306    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe9636f-eab1-4c19-a74b-8f24b051449f-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.774058 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.771410    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hj86\" (UniqueName: \"kubernetes.io/projected/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-api-access-5hj86\") pod \"isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.775309 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.775278    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:32.897068 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.897032    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:32.952132 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:32.952095    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 23:23:33.178306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:33.178282    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"]
Sep 17 23:23:33.180259 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:23:33.180236    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe9636f_eab1_4c19_a74b_8f24b051449f.slice/crio-e26014240c62c9c29f8bd8f882b5a7b1e731ed63554202c0aa3c895d66064ca0 WatchSource:0}: Error finding container e26014240c62c9c29f8bd8f882b5a7b1e731ed63554202c0aa3c895d66064ca0: Status 404 returned error can't find the container with id e26014240c62c9c29f8bd8f882b5a7b1e731ed63554202c0aa3c895d66064ca0
Sep 17 23:23:33.278283 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:33.278255    2571 generic.go:356] "Generic (PLEG): container finished" podID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerID="eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f" exitCode=2
Sep 17 23:23:33.278435 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:33.278348    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerDied","Data":"eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f"}
Sep 17 23:23:33.280018 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:33.279988    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerStarted","Data":"ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498"}
Sep 17 23:23:33.280113 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:33.280056    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerStarted","Data":"e26014240c62c9c29f8bd8f882b5a7b1e731ed63554202c0aa3c895d66064ca0"}
Sep 17 23:23:37.299911 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:37.299884    2571 generic.go:356] "Generic (PLEG): container finished" podID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerID="588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943" exitCode=0
Sep 17 23:23:37.300252 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:37.299954    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerDied","Data":"588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943"}
Sep 17 23:23:37.301539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:37.301520    2571 generic.go:356] "Generic (PLEG): container finished" podID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerID="ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498" exitCode=0
Sep 17 23:23:37.301639 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:37.301587    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerDied","Data":"ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498"}
Sep 17 23:23:37.951587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:37.951547    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 23:23:37.956910 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:37.956881    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:23:37.957350 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:37.957308    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:38.308056 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:38.307982    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerStarted","Data":"84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653"}
Sep 17 23:23:38.308512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:38.308059    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerStarted","Data":"0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899"}
Sep 17 23:23:38.308512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:38.308093    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerStarted","Data":"9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057"}
Sep 17 23:23:38.308512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:38.308424    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:38.308675 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:38.308591    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:38.309706 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:38.309684    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:23:38.334598 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:38.334560    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podStartSLOduration=2.375144023 podStartE2EDuration="6.334548227s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.959404204s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:23:38.332439029 +0000 UTC m=+742.793356762" watchObservedRunningTime="2026-09-17 23:23:38.334548227 +0000 UTC m=+742.795465936"
Sep 17 23:23:39.313071 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:39.313028    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:23:39.313550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:39.313189    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:39.314310 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:39.314286    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:40.317367 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:40.317303    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:23:40.317812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:40.317683    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:42.951595 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:42.951544    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 23:23:42.951943 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:42.951681    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:23:45.322414 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:45.322378    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:23:45.323019 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:45.322989    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:23:45.323509 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:45.323487    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:47.952474 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:47.952428    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 23:23:47.956757 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:47.956734    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:23:47.958260 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:47.958229    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:52.952342 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:52.952284    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 23:23:53.111213 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:53.111179    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:55.323474 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:55.323428    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:23:55.323956 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:55.323932    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:57.951681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:57.951636    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Sep 17 23:23:57.956975 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:57.956950    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Sep 17 23:23:57.957112 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:57.957096    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:23:57.957429 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:57.957403    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:23:57.957521 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:57.957509    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:23:58.246053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:58.245969    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:23:59.059270 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:23:59.059223    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:02.562879 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.562854    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:24:02.684479 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.684423    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kserve-provision-location\") pod \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") "
Sep 17 23:24:02.684479 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.684476    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfmcr\" (UniqueName: \"kubernetes.io/projected/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-api-access-hfmcr\") pod \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") "
Sep 17 23:24:02.684639 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.684523    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b44bc1-044c-4a0e-810d-84ce8da88d75-proxy-tls\") pod \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") "
Sep 17 23:24:02.684639 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.684568    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-cabundle-cert\") pod \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") "
Sep 17 23:24:02.684639 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.684597    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-rbac-proxy-sar-config\") pod \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\" (UID: \"a1b44bc1-044c-4a0e-810d-84ce8da88d75\") "
Sep 17 23:24:02.684789 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.684742    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1b44bc1-044c-4a0e-810d-84ce8da88d75" (UID: "a1b44bc1-044c-4a0e-810d-84ce8da88d75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:24:02.684996 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.684969    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1b44bc1-044c-4a0e-810d-84ce8da88d75" (UID: "a1b44bc1-044c-4a0e-810d-84ce8da88d75"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:02.685069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.684971    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1b44bc1-044c-4a0e-810d-84ce8da88d75" (UID: "a1b44bc1-044c-4a0e-810d-84ce8da88d75"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:24:02.686606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.686585    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b44bc1-044c-4a0e-810d-84ce8da88d75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1b44bc1-044c-4a0e-810d-84ce8da88d75" (UID: "a1b44bc1-044c-4a0e-810d-84ce8da88d75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:24:02.686775 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.686755    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-api-access-hfmcr" (OuterVolumeSpecName: "kube-api-access-hfmcr") pod "a1b44bc1-044c-4a0e-810d-84ce8da88d75" (UID: "a1b44bc1-044c-4a0e-810d-84ce8da88d75"). InnerVolumeSpecName "kube-api-access-hfmcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:24:02.785761 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.785737    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1b44bc1-044c-4a0e-810d-84ce8da88d75-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:24:02.785761 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.785758    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:24:02.785885 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.785767    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:24:02.785885 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.785776    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:24:02.785885 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:02.785785    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfmcr\" (UniqueName: \"kubernetes.io/projected/a1b44bc1-044c-4a0e-810d-84ce8da88d75-kube-api-access-hfmcr\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:24:03.417431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.417401    2571 generic.go:356] "Generic (PLEG): container finished" podID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerID="f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab" exitCode=0
Sep 17 23:24:03.417584 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.417460    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"
Sep 17 23:24:03.417584 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.417481    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerDied","Data":"f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab"}
Sep 17 23:24:03.417584 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.417546    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f" event={"ID":"a1b44bc1-044c-4a0e-810d-84ce8da88d75","Type":"ContainerDied","Data":"1d6a7242a49ed4dace36a58e77b16b941ab10d7a7cffc2c7674b6c3cdbbf636b"}
Sep 17 23:24:03.417584 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.417578    2571 scope.go:117] "RemoveContainer" containerID="f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab"
Sep 17 23:24:03.425797 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.425779    2571 scope.go:117] "RemoveContainer" containerID="eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f"
Sep 17 23:24:03.433076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.433058    2571 scope.go:117] "RemoveContainer" containerID="588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943"
Sep 17 23:24:03.440179 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.440158    2571 scope.go:117] "RemoveContainer" containerID="366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606"
Sep 17 23:24:03.442562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.442535    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"]
Sep 17 23:24:03.446277 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.446256    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-db7d7fc4c-xzj6f"]
Sep 17 23:24:03.447760 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.447746    2571 scope.go:117] "RemoveContainer" containerID="f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab"
Sep 17 23:24:03.448043 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:24:03.448028    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab\": container with ID starting with f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab not found: ID does not exist" containerID="f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab"
Sep 17 23:24:03.448088 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.448050    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab"} err="failed to get container status \"f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab\": rpc error: code = NotFound desc = could not find container \"f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab\": container with ID starting with f99c76a44b906051f2c900a063d17da292b94796aaa08c8ac4cd09ba00773bab not found: ID does not exist"
Sep 17 23:24:03.448088 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.448066    2571 scope.go:117] "RemoveContainer" containerID="eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f"
Sep 17 23:24:03.448310 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:24:03.448295    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f\": container with ID starting with eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f not found: ID does not exist" containerID="eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f"
Sep 17 23:24:03.448401 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.448313    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f"} err="failed to get container status \"eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f\": rpc error: code = NotFound desc = could not find container \"eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f\": container with ID starting with eb174c06199bef18bb3980ef7bdbafabeb82c5ea2cb28892d2bf9783938e7b9f not found: ID does not exist"
Sep 17 23:24:03.448401 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.448345    2571 scope.go:117] "RemoveContainer" containerID="588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943"
Sep 17 23:24:03.448562 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:24:03.448548    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943\": container with ID starting with 588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943 not found: ID does not exist" containerID="588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943"
Sep 17 23:24:03.448601 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.448564    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943"} err="failed to get container status \"588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943\": rpc error: code = NotFound desc = could not find container \"588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943\": container with ID starting with 588170ebc1783a840aeb2bde3392d537f2920965bdd85fc432ab33ff888c2943 not found: ID does not exist"
Sep 17 23:24:03.448601 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.448575    2571 scope.go:117] "RemoveContainer" containerID="366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606"
Sep 17 23:24:03.448800 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:24:03.448780    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606\": container with ID starting with 366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606 not found: ID does not exist" containerID="366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606"
Sep 17 23:24:03.448838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:03.448808    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606"} err="failed to get container status \"366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606\": rpc error: code = NotFound desc = could not find container \"366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606\": container with ID starting with 366619f0a7a7913204961d79ea4569810f6e9d0b1f7bde78b4ff03a970459606 not found: ID does not exist"
Sep 17 23:24:04.057953 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:04.057919    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" path="/var/lib/kubelet/pods/a1b44bc1-044c-4a0e-810d-84ce8da88d75/volumes"
Sep 17 23:24:05.323066 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:05.323017    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:24:05.323652 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:05.323627    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:09.468390 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:09.468348    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:09.567604 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:09.567577    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:15.323563 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:15.323521    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:24:15.324027 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:15.323964    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:25.323539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:25.323484    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:24:25.324047 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:25.324020    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:35.323367 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:35.323302    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:24:35.323758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:35.323690    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:45.323157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:45.323058    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:24:45.323701 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:45.323586    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:24:53.377158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:53.377122    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:53.478002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:53.477966    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:24:54.059066 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:54.059042    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:24:54.059214 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:24:54.059116    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:25:07.610766 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.610718    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"]
Sep 17 23:25:07.611254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.611038    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" containerID="cri-o://9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057" gracePeriod=30
Sep 17 23:25:07.611254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.611059    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" containerID="cri-o://84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653" gracePeriod=30
Sep 17 23:25:07.611254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.611089    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy" containerID="cri-o://0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899" gracePeriod=30
Sep 17 23:25:07.638478 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.638445    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:07.699936 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.699909    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"]
Sep 17 23:25:07.700241 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700228    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container"
Sep 17 23:25:07.700284 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700243    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container"
Sep 17 23:25:07.700284 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700255    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy"
Sep 17 23:25:07.700284 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700264    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy"
Sep 17 23:25:07.700284 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700275    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="storage-initializer"
Sep 17 23:25:07.700284 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700281    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="storage-initializer"
Sep 17 23:25:07.700470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700299    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent"
Sep 17 23:25:07.700470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700305    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent"
Sep 17 23:25:07.700470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700378    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kube-rbac-proxy"
Sep 17 23:25:07.700470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700386    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="kserve-container"
Sep 17 23:25:07.700470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.700393    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1b44bc1-044c-4a0e-810d-84ce8da88d75" containerName="agent"
Sep 17 23:25:07.703595 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.703580    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.706272 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.706251    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 23:25:07.706403 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.706283    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 23:25:07.712061 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.712042    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"]
Sep 17 23:25:07.740083 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.740052    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:07.842605 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.842579    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.842715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.842631    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnrbv\" (UniqueName: \"kubernetes.io/projected/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-api-access-qnrbv\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.842715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.842661    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.943624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.943567    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.943624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.943607    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnrbv\" (UniqueName: \"kubernetes.io/projected/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-api-access-qnrbv\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.943624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.943623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.944205 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.944176    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.945235 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.945213    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:07.946107 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.946090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:07.957566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:07.957543    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnrbv\" (UniqueName: \"kubernetes.io/projected/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-api-access-qnrbv\") pod \"message-dumper-predictor-6575bcfdfd-wqg4v\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:08.014370 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:08.014347    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:08.287504 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:08.287481    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"]
Sep 17 23:25:08.289784 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:25:08.289754    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0ea5f3_ba61_4d3c_80ac_e871b53ec0e2.slice/crio-9da6b896779ce0c32f72a1dd83513fb8c6137dd6fae6af2f0175d62356ad887a WatchSource:0}: Error finding container 9da6b896779ce0c32f72a1dd83513fb8c6137dd6fae6af2f0175d62356ad887a: Status 404 returned error can't find the container with id 9da6b896779ce0c32f72a1dd83513fb8c6137dd6fae6af2f0175d62356ad887a
Sep 17 23:25:08.291660 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:08.291643    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:25:08.702530 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:08.702493    2571 generic.go:356] "Generic (PLEG): container finished" podID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerID="0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899" exitCode=2
Sep 17 23:25:08.702909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:08.702580    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerDied","Data":"0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899"}
Sep 17 23:25:08.703817 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:08.703792    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" event={"ID":"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2","Type":"ContainerStarted","Data":"9da6b896779ce0c32f72a1dd83513fb8c6137dd6fae6af2f0175d62356ad887a"}
Sep 17 23:25:09.709946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:09.709924    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" event={"ID":"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2","Type":"ContainerStarted","Data":"a077639d67e7f7725113888c92e731988a3c30793d7f7596465a1092dc247960"}
Sep 17 23:25:10.317583 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:10.317540    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 23:25:10.716244 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:10.716211    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" event={"ID":"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2","Type":"ContainerStarted","Data":"cf549fd6f8b959381e45bfe2b534f149ef183ef00986470ca13e5b1aa1d6ddcb"}
Sep 17 23:25:10.716713 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:10.716343    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:10.736090 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:10.736035    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" podStartSLOduration=2.380470118 podStartE2EDuration="3.736015091s" totalImagesPullingTime="1.355544973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:10.733379722 +0000 UTC m=+835.194297452" watchObservedRunningTime="2026-09-17 23:25:10.736015091 +0000 UTC m=+835.196932824"
Sep 17 23:25:11.721313 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:11.721284    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:11.723178 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:11.723155    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:12.728278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:12.728244    2571 generic.go:356] "Generic (PLEG): container finished" podID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerID="9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057" exitCode=0
Sep 17 23:25:12.728806 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:12.728366    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerDied","Data":"9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057"}
Sep 17 23:25:14.054050 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:14.054005    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:25:14.054931 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:14.054902    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:15.318316 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:15.318273    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 23:25:18.736443 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:18.736414    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:25:20.317403 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:20.317365    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 23:25:20.317781 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:20.317516    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:25:24.054242 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:24.054199    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:25:24.054698 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:24.054602    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:25.317830 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:25.317779    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 23:25:27.726511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.726462    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:27.757452 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.757421    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"]
Sep 17 23:25:27.760899 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.760883    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.763492 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.763472    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 23:25:27.763580 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.763527    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 23:25:27.771109 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.771090    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"]
Sep 17 23:25:27.895294 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.895259    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzl2j\" (UniqueName: \"kubernetes.io/projected/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-api-access-nzl2j\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.895434 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.895305    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.895434 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.895349    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-cabundle-cert\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.895554 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.895459    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.895554 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.895495    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772765a5-3682-4b88-adcf-ac4a5527e8d4-kserve-provision-location\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.996407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.996359    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.996407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.996388    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772765a5-3682-4b88-adcf-ac4a5527e8d4-kserve-provision-location\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.996552 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.996413    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzl2j\" (UniqueName: \"kubernetes.io/projected/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-api-access-nzl2j\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.996552 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.996443    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.996552 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.996467    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-cabundle-cert\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.996552 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:25:27.996504    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 17 23:25:27.996753 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:25:27.996557    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls podName:772765a5-3682-4b88-adcf-ac4a5527e8d4 nodeName:}" failed. No retries permitted until 2026-09-17 23:25:28.496540474 +0000 UTC m=+852.957458198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls") pod "isvc-logger-predictor-6465ffc5dd-vdqr7" (UID: "772765a5-3682-4b88-adcf-ac4a5527e8d4") : secret "isvc-logger-predictor-serving-cert" not found
Sep 17 23:25:27.996810 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.996788    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772765a5-3682-4b88-adcf-ac4a5527e8d4-kserve-provision-location\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.997060 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.997043    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-cabundle-cert\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:27.997095 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:27.997060    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:28.005783 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:28.005758    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzl2j\" (UniqueName: \"kubernetes.io/projected/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-api-access-nzl2j\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:28.105922 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:28.105894    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:28.501005 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:28.500977    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:28.503584 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:28.503557    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls\") pod \"isvc-logger-predictor-6465ffc5dd-vdqr7\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:28.671254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:28.671228    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:28.800992 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:28.800894    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:28.949473 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:28.949447    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"]
Sep 17 23:25:28.952690 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:25:28.952666    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772765a5_3682_4b88_adcf_ac4a5527e8d4.slice/crio-fd5ceca8a9d4fac848e1206302e87d3c52b5b51f5cf93505528701149c75f3ec WatchSource:0}: Error finding container fd5ceca8a9d4fac848e1206302e87d3c52b5b51f5cf93505528701149c75f3ec: Status 404 returned error can't find the container with id fd5ceca8a9d4fac848e1206302e87d3c52b5b51f5cf93505528701149c75f3ec
Sep 17 23:25:29.802953 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:29.802922    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerStarted","Data":"7223a0796ffee5abe38090467b827ec034a68684699c4d6bb25caa0f269afa36"}
Sep 17 23:25:29.803318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:29.802978    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerStarted","Data":"fd5ceca8a9d4fac848e1206302e87d3c52b5b51f5cf93505528701149c75f3ec"}
Sep 17 23:25:30.317924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:30.317874    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 23:25:32.816926 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:32.816898    2571 generic.go:356] "Generic (PLEG): container finished" podID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerID="7223a0796ffee5abe38090467b827ec034a68684699c4d6bb25caa0f269afa36" exitCode=0
Sep 17 23:25:32.817213 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:32.816960    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerDied","Data":"7223a0796ffee5abe38090467b827ec034a68684699c4d6bb25caa0f269afa36"}
Sep 17 23:25:33.823557 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:33.823520    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerStarted","Data":"6f8816592ccac585ffa48581e21a66f06818ac7ef983a830e911c7404193b648"}
Sep 17 23:25:33.823909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:33.823587    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerStarted","Data":"91ab389aacf97c90a3f16897408aa9bbdbf3d1b8cea5c53bef1772f004a23732"}
Sep 17 23:25:33.823909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:33.823613    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerStarted","Data":"80ab60ae369ebbaf88d7547b249998923284acf8da2c7a01bbc63d4fded8c46b"}
Sep 17 23:25:33.823909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:33.823900    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:33.824081 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:33.824063    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:33.825169 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:33.825146    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:25:33.857619 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:33.857580    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podStartSLOduration=3.104102521 podStartE2EDuration="6.857566716s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.753464195s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:33.855250594 +0000 UTC m=+858.316168329" watchObservedRunningTime="2026-09-17 23:25:33.857566716 +0000 UTC m=+858.318484448"
Sep 17 23:25:33.940256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:33.940224    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:34.054194 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:34.054155    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:5000: connect: connection refused"
Sep 17 23:25:34.054484 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:34.054460    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:34.058294 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:34.058278    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:25:34.058392 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:34.058381    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:25:34.215153 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:34.215129    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:25:34.828208 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:34.828168    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:25:34.828663 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:34.828293    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:34.829467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:34.829444    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:35.317954 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:35.317918    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Sep 17 23:25:35.833117 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:35.833084    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:25:35.833630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:35.833608    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:37.757054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.757031    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:25:37.843269 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.843234    2571 generic.go:356] "Generic (PLEG): container finished" podID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerID="84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653" exitCode=0
Sep 17 23:25:37.843393 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.843308    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"
Sep 17 23:25:37.843393 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.843306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerDied","Data":"84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653"}
Sep 17 23:25:37.843393 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.843385    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2" event={"ID":"cfe9636f-eab1-4c19-a74b-8f24b051449f","Type":"ContainerDied","Data":"e26014240c62c9c29f8bd8f882b5a7b1e731ed63554202c0aa3c895d66064ca0"}
Sep 17 23:25:37.843490 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.843418    2571 scope.go:117] "RemoveContainer" containerID="84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653"
Sep 17 23:25:37.851394 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.851380    2571 scope.go:117] "RemoveContainer" containerID="0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899"
Sep 17 23:25:37.858963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.858946    2571 scope.go:117] "RemoveContainer" containerID="9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057"
Sep 17 23:25:37.866003 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.865968    2571 scope.go:117] "RemoveContainer" containerID="ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498"
Sep 17 23:25:37.866077 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.866028    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-rbac-proxy-sar-config\") pod \"cfe9636f-eab1-4c19-a74b-8f24b051449f\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") "
Sep 17 23:25:37.866077 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.866061    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe9636f-eab1-4c19-a74b-8f24b051449f-kserve-provision-location\") pod \"cfe9636f-eab1-4c19-a74b-8f24b051449f\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") "
Sep 17 23:25:37.866190 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.866090    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-cabundle-cert\") pod \"cfe9636f-eab1-4c19-a74b-8f24b051449f\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") "
Sep 17 23:25:37.866190 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.866152    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe9636f-eab1-4c19-a74b-8f24b051449f-proxy-tls\") pod \"cfe9636f-eab1-4c19-a74b-8f24b051449f\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") "
Sep 17 23:25:37.866489 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.866214    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hj86\" (UniqueName: \"kubernetes.io/projected/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-api-access-5hj86\") pod \"cfe9636f-eab1-4c19-a74b-8f24b051449f\" (UID: \"cfe9636f-eab1-4c19-a74b-8f24b051449f\") "
Sep 17 23:25:37.866489 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.866385    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe9636f-eab1-4c19-a74b-8f24b051449f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfe9636f-eab1-4c19-a74b-8f24b051449f" (UID: "cfe9636f-eab1-4c19-a74b-8f24b051449f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:37.866489 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.866417    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cfe9636f-eab1-4c19-a74b-8f24b051449f" (UID: "cfe9636f-eab1-4c19-a74b-8f24b051449f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:37.866646 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.866521    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfe9636f-eab1-4c19-a74b-8f24b051449f" (UID: "cfe9636f-eab1-4c19-a74b-8f24b051449f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:25:37.868369 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.868341    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfe9636f-eab1-4c19-a74b-8f24b051449f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cfe9636f-eab1-4c19-a74b-8f24b051449f" (UID: "cfe9636f-eab1-4c19-a74b-8f24b051449f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:25:37.868476 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.868453    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-api-access-5hj86" (OuterVolumeSpecName: "kube-api-access-5hj86") pod "cfe9636f-eab1-4c19-a74b-8f24b051449f" (UID: "cfe9636f-eab1-4c19-a74b-8f24b051449f"). InnerVolumeSpecName "kube-api-access-5hj86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:37.881813 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.881796    2571 scope.go:117] "RemoveContainer" containerID="84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653"
Sep 17 23:25:37.882088 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:25:37.882067    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653\": container with ID starting with 84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653 not found: ID does not exist" containerID="84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653"
Sep 17 23:25:37.882183 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.882093    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653"} err="failed to get container status \"84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653\": rpc error: code = NotFound desc = could not find container \"84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653\": container with ID starting with 84d8e36ea7139d00352c2105e6e3a7d8bc8e382bb0cdf9b3b6dd7e30ded4b653 not found: ID does not exist"
Sep 17 23:25:37.882183 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.882109    2571 scope.go:117] "RemoveContainer" containerID="0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899"
Sep 17 23:25:37.882384 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:25:37.882367    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899\": container with ID starting with 0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899 not found: ID does not exist" containerID="0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899"
Sep 17 23:25:37.882429 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.882387    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899"} err="failed to get container status \"0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899\": rpc error: code = NotFound desc = could not find container \"0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899\": container with ID starting with 0c9610af964186e8163e69ad79ab1944801aacbc031433b25481e5ccc8a9b899 not found: ID does not exist"
Sep 17 23:25:37.882429 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.882398    2571 scope.go:117] "RemoveContainer" containerID="9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057"
Sep 17 23:25:37.882613 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:25:37.882599    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057\": container with ID starting with 9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057 not found: ID does not exist" containerID="9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057"
Sep 17 23:25:37.882661 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.882614    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057"} err="failed to get container status \"9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057\": rpc error: code = NotFound desc = could not find container \"9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057\": container with ID starting with 9e11127ddb9cc42c1a7091f9b5f7ea7f4e2a6ade6411203675f4bcf9d2381057 not found: ID does not exist"
Sep 17 23:25:37.882661 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.882624    2571 scope.go:117] "RemoveContainer" containerID="ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498"
Sep 17 23:25:37.882856 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:25:37.882843    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498\": container with ID starting with ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498 not found: ID does not exist" containerID="ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498"
Sep 17 23:25:37.882909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.882857    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498"} err="failed to get container status \"ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498\": rpc error: code = NotFound desc = could not find container \"ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498\": container with ID starting with ac081adba167cc14f95080527227a27d38b74d5a0a78c82737b9f6019200e498 not found: ID does not exist"
Sep 17 23:25:37.966672 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.966653    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfe9636f-eab1-4c19-a74b-8f24b051449f-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:25:37.966672 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.966672    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hj86\" (UniqueName: \"kubernetes.io/projected/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-api-access-5hj86\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:25:37.966840 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.966681    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:25:37.966840 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.966692    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfe9636f-eab1-4c19-a74b-8f24b051449f-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:25:37.966840 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:37.966703    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfe9636f-eab1-4c19-a74b-8f24b051449f-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:25:38.160202 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:38.160142    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"]
Sep 17 23:25:38.163296 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:38.163268    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7778d45fd9-bmtt2"]
Sep 17 23:25:40.057715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:40.057682    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" path="/var/lib/kubelet/pods/cfe9636f-eab1-4c19-a74b-8f24b051449f/volumes"
Sep 17 23:25:40.837607 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:40.837584    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:25:40.838189 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:40.838156    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:25:40.838428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:40.838404    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:25:50.838069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:50.838025    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:25:50.838578 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:25:50.838509    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:00.838879 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:00.838830    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:26:00.839300 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:00.839275    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:10.838559 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:10.838464    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:26:10.839033 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:10.839002    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:16.024062 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:16.024031    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:26:16.025499 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:16.025473    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:26:20.838609 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:20.838572    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:26:20.839048 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:20.839025    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:30.838672 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:30.838627    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:26:30.841153 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:30.839065    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:40.838599 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:40.838559    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:26:40.839024 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:40.838995    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:26:50.839464 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:50.839436    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:26:50.839864 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:26:50.839533    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:27:02.788103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.788065    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-wqg4v_2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2/kserve-container/0.log"
Sep 17 23:27:02.941936 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.941900    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"]
Sep 17 23:27:02.942239 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.942211    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" containerID="cri-o://80ab60ae369ebbaf88d7547b249998923284acf8da2c7a01bbc63d4fded8c46b" gracePeriod=30
Sep 17 23:27:02.942407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.942299    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy" containerID="cri-o://91ab389aacf97c90a3f16897408aa9bbdbf3d1b8cea5c53bef1772f004a23732" gracePeriod=30
Sep 17 23:27:02.942578 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.942304    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" containerID="cri-o://6f8816592ccac585ffa48581e21a66f06818ac7ef983a830e911c7404193b648" gracePeriod=30
Sep 17 23:27:02.965611 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.965584    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:02.973362 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.973312    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"]
Sep 17 23:27:02.973930 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.973896    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerName="kserve-container" containerID="cri-o://a077639d67e7f7725113888c92e731988a3c30793d7f7596465a1092dc247960" gracePeriod=30
Sep 17 23:27:02.974204 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:02.974155    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerName="kube-rbac-proxy" containerID="cri-o://cf549fd6f8b959381e45bfe2b534f149ef183ef00986470ca13e5b1aa1d6ddcb" gracePeriod=30
Sep 17 23:27:03.067525 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.067460    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:03.218532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.218497    2571 generic.go:356] "Generic (PLEG): container finished" podID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerID="cf549fd6f8b959381e45bfe2b534f149ef183ef00986470ca13e5b1aa1d6ddcb" exitCode=2
Sep 17 23:27:03.218532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.218520    2571 generic.go:356] "Generic (PLEG): container finished" podID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerID="a077639d67e7f7725113888c92e731988a3c30793d7f7596465a1092dc247960" exitCode=2
Sep 17 23:27:03.218712 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.218572    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" event={"ID":"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2","Type":"ContainerDied","Data":"cf549fd6f8b959381e45bfe2b534f149ef183ef00986470ca13e5b1aa1d6ddcb"}
Sep 17 23:27:03.218712 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.218631    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" event={"ID":"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2","Type":"ContainerDied","Data":"a077639d67e7f7725113888c92e731988a3c30793d7f7596465a1092dc247960"}
Sep 17 23:27:03.218712 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.218651    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v" event={"ID":"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2","Type":"ContainerDied","Data":"9da6b896779ce0c32f72a1dd83513fb8c6137dd6fae6af2f0175d62356ad887a"}
Sep 17 23:27:03.218712 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.218680    2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9da6b896779ce0c32f72a1dd83513fb8c6137dd6fae6af2f0175d62356ad887a"
Sep 17 23:27:03.221167 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.221148    2571 generic.go:356] "Generic (PLEG): container finished" podID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerID="91ab389aacf97c90a3f16897408aa9bbdbf3d1b8cea5c53bef1772f004a23732" exitCode=2
Sep 17 23:27:03.221278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.221208    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerDied","Data":"91ab389aacf97c90a3f16897408aa9bbdbf3d1b8cea5c53bef1772f004a23732"}
Sep 17 23:27:03.227295 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.227278    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:27:03.338107 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.338072    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:03.367310 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.367289    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-rbac-proxy-sar-config\") pod \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") "
Sep 17 23:27:03.367441 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.367357    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-proxy-tls\") pod \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") "
Sep 17 23:27:03.367441 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.367409    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnrbv\" (UniqueName: \"kubernetes.io/projected/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-api-access-qnrbv\") pod \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\" (UID: \"2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2\") "
Sep 17 23:27:03.367694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.367664    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" (UID: "2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:03.369467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.369447    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" (UID: "2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:03.369542 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.369447    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-api-access-qnrbv" (OuterVolumeSpecName: "kube-api-access-qnrbv") pod "2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" (UID: "2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2"). InnerVolumeSpecName "kube-api-access-qnrbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:03.468187 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.468162    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:27:03.468187 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.468183    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:27:03.468305 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:03.468192    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnrbv\" (UniqueName: \"kubernetes.io/projected/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2-kube-api-access-qnrbv\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:27:04.225399 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:04.225312    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"
Sep 17 23:27:04.246426 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:04.246390    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"]
Sep 17 23:27:04.250708 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:04.250681    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-wqg4v"]
Sep 17 23:27:05.834066 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:05.834022    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 23:27:06.058724 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:06.058692    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" path="/var/lib/kubelet/pods/2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2/volumes"
Sep 17 23:27:07.239159 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:07.239096    2571 generic.go:356] "Generic (PLEG): container finished" podID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerID="80ab60ae369ebbaf88d7547b249998923284acf8da2c7a01bbc63d4fded8c46b" exitCode=0
Sep 17 23:27:07.239159 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:07.239153    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerDied","Data":"80ab60ae369ebbaf88d7547b249998923284acf8da2c7a01bbc63d4fded8c46b"}
Sep 17 23:27:10.833760 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:10.833711    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 23:27:10.838040 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:10.837997    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:27:10.838375 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:10.838316    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:27:15.833704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:15.833656    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 23:27:15.834158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:15.833825    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:27:16.613365 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:16.613314    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:20.833680 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:20.833628    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 23:27:20.838983 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:20.838954    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:27:20.839410 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:20.839385    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:27:21.760425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.760391    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:21.780891 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.780862    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"]
Sep 17 23:27:21.781179 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781168    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container"
Sep 17 23:27:21.781222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781181    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container"
Sep 17 23:27:21.781222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781192    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy"
Sep 17 23:27:21.781222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781197    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy"
Sep 17 23:27:21.781222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781203    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerName="kube-rbac-proxy"
Sep 17 23:27:21.781222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781209    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerName="kube-rbac-proxy"
Sep 17 23:27:21.781222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781223    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781229    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781236    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerName="kserve-container"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781241    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerName="kserve-container"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781250    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="storage-initializer"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781256    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="storage-initializer"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781308    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kserve-container"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781316    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerName="kserve-container"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781338    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="agent"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781346    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe9636f-eab1-4c19-a74b-8f24b051449f" containerName="kube-rbac-proxy"
Sep 17 23:27:21.781431 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.781351    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0ea5f3-ba61-4d3c-80ac-e871b53ec0e2" containerName="kube-rbac-proxy"
Sep 17 23:27:21.784447 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.784432    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.787166 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.787149    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 23:27:21.787166 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.787161    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 23:27:21.793728 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.793708    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"]
Sep 17 23:27:21.801919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.801888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjm78\" (UniqueName: \"kubernetes.io/projected/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-api-access-wjm78\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.802016 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.801931    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.802016 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.802000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.802127 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.802105    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.802178 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.802161    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.903160 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.903127    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.903160 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.903159    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.903600 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.903186    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjm78\" (UniqueName: \"kubernetes.io/projected/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-api-access-wjm78\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.903600 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.903206    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.903600 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.903233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.903600 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:27:21.903275    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 23:27:21.903600 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:27:21.903379    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls podName:9805ee7e-7980-4c97-b1f3-7d09308c5bce nodeName:}" failed. No retries permitted until 2026-09-17 23:27:22.403358682 +0000 UTC m=+966.864276394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" (UID: "9805ee7e-7980-4c97-b1f3-7d09308c5bce") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Sep 17 23:27:21.903600 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.903593    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.903949 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.903920    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.904027 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.903999    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:21.912170 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:21.912152    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjm78\" (UniqueName: \"kubernetes.io/projected/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-api-access-wjm78\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:22.119227 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:22.119197    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:27:22.406500 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:22.406438    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:22.409004 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:22.408976    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:22.695269 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:22.695216    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:22.975761 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:22.975673    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"]
Sep 17 23:27:22.981719 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:27:22.981690    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9805ee7e_7980_4c97_b1f3_7d09308c5bce.slice/crio-dcacc9b368a0df64421e698573f5ff6f172a798cb97a3c8259f65a6e35bdc565 WatchSource:0}: Error finding container dcacc9b368a0df64421e698573f5ff6f172a798cb97a3c8259f65a6e35bdc565: Status 404 returned error can't find the container with id dcacc9b368a0df64421e698573f5ff6f172a798cb97a3c8259f65a6e35bdc565
Sep 17 23:27:23.315919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:23.315837    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" event={"ID":"9805ee7e-7980-4c97-b1f3-7d09308c5bce","Type":"ContainerStarted","Data":"88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77"}
Sep 17 23:27:23.315919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:23.315900    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" event={"ID":"9805ee7e-7980-4c97-b1f3-7d09308c5bce","Type":"ContainerStarted","Data":"dcacc9b368a0df64421e698573f5ff6f172a798cb97a3c8259f65a6e35bdc565"}
Sep 17 23:27:25.833537 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:25.833492    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 23:27:27.333782 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:27.333754    2571 generic.go:356] "Generic (PLEG): container finished" podID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerID="88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77" exitCode=0
Sep 17 23:27:27.334188 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:27.333835    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" event={"ID":"9805ee7e-7980-4c97-b1f3-7d09308c5bce","Type":"ContainerDied","Data":"88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77"}
Sep 17 23:27:30.833835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:30.833563    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Sep 17 23:27:30.839177 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:30.839135    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Sep 17 23:27:30.839346 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:30.839278    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:27:30.839732 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:30.839662    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:27:30.839855 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:30.839789    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:27:33.388469 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:33.388424    2571 generic.go:356] "Generic (PLEG): container finished" podID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerID="6f8816592ccac585ffa48581e21a66f06818ac7ef983a830e911c7404193b648" exitCode=0
Sep 17 23:27:33.388882 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:33.388494    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerDied","Data":"6f8816592ccac585ffa48581e21a66f06818ac7ef983a830e911c7404193b648"}
Sep 17 23:27:36.834533 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:36.834473    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 23:27:36.996934 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:36.996894    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:27:37.020263 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.020230    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzl2j\" (UniqueName: \"kubernetes.io/projected/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-api-access-nzl2j\") pod \"772765a5-3682-4b88-adcf-ac4a5527e8d4\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") "
Sep 17 23:27:37.020437 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.020288    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772765a5-3682-4b88-adcf-ac4a5527e8d4-kserve-provision-location\") pod \"772765a5-3682-4b88-adcf-ac4a5527e8d4\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") "
Sep 17 23:27:37.020437 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.020344    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-rbac-proxy-sar-config\") pod \"772765a5-3682-4b88-adcf-ac4a5527e8d4\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") "
Sep 17 23:27:37.020437 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.020372    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-cabundle-cert\") pod \"772765a5-3682-4b88-adcf-ac4a5527e8d4\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") "
Sep 17 23:27:37.020437 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.020434    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls\") pod \"772765a5-3682-4b88-adcf-ac4a5527e8d4\" (UID: \"772765a5-3682-4b88-adcf-ac4a5527e8d4\") "
Sep 17 23:27:37.020857 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.020706    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/772765a5-3682-4b88-adcf-ac4a5527e8d4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "772765a5-3682-4b88-adcf-ac4a5527e8d4" (UID: "772765a5-3682-4b88-adcf-ac4a5527e8d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:37.020992 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.020899    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "772765a5-3682-4b88-adcf-ac4a5527e8d4" (UID: "772765a5-3682-4b88-adcf-ac4a5527e8d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:37.021069 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.021035    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "772765a5-3682-4b88-adcf-ac4a5527e8d4" (UID: "772765a5-3682-4b88-adcf-ac4a5527e8d4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:27:37.023516 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.023486    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-api-access-nzl2j" (OuterVolumeSpecName: "kube-api-access-nzl2j") pod "772765a5-3682-4b88-adcf-ac4a5527e8d4" (UID: "772765a5-3682-4b88-adcf-ac4a5527e8d4"). InnerVolumeSpecName "kube-api-access-nzl2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:37.024746 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.024721    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "772765a5-3682-4b88-adcf-ac4a5527e8d4" (UID: "772765a5-3682-4b88-adcf-ac4a5527e8d4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:37.121832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.121689    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:27:37.121832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.121722    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/772765a5-3682-4b88-adcf-ac4a5527e8d4-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:27:37.121832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.121738    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/772765a5-3682-4b88-adcf-ac4a5527e8d4-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:27:37.121832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.121751    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzl2j\" (UniqueName: \"kubernetes.io/projected/772765a5-3682-4b88-adcf-ac4a5527e8d4-kube-api-access-nzl2j\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:27:37.121832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.121765    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/772765a5-3682-4b88-adcf-ac4a5527e8d4-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:27:37.415318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.415204    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7" event={"ID":"772765a5-3682-4b88-adcf-ac4a5527e8d4","Type":"ContainerDied","Data":"fd5ceca8a9d4fac848e1206302e87d3c52b5b51f5cf93505528701149c75f3ec"}
Sep 17 23:27:37.415318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.415265    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"
Sep 17 23:27:37.415318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.415301    2571 scope.go:117] "RemoveContainer" containerID="6f8816592ccac585ffa48581e21a66f06818ac7ef983a830e911c7404193b648"
Sep 17 23:27:37.443681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.443643    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"]
Sep 17 23:27:37.446843 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:37.446814    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6465ffc5dd-vdqr7"]
Sep 17 23:27:38.058661 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:38.058625    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" path="/var/lib/kubelet/pods/772765a5-3682-4b88-adcf-ac4a5527e8d4/volumes"
Sep 17 23:27:38.587813 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:38.587603    2571 scope.go:117] "RemoveContainer" containerID="91ab389aacf97c90a3f16897408aa9bbdbf3d1b8cea5c53bef1772f004a23732"
Sep 17 23:27:38.597388 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:38.597367    2571 scope.go:117] "RemoveContainer" containerID="80ab60ae369ebbaf88d7547b249998923284acf8da2c7a01bbc63d4fded8c46b"
Sep 17 23:27:38.606290 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:38.606273    2571 scope.go:117] "RemoveContainer" containerID="7223a0796ffee5abe38090467b827ec034a68684699c4d6bb25caa0f269afa36"
Sep 17 23:27:49.480201 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:49.480120    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" event={"ID":"9805ee7e-7980-4c97-b1f3-7d09308c5bce","Type":"ContainerStarted","Data":"bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c"}
Sep 17 23:27:49.480624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:49.480223    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" event={"ID":"9805ee7e-7980-4c97-b1f3-7d09308c5bce","Type":"ContainerStarted","Data":"ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e"}
Sep 17 23:27:49.480624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:49.480457    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:49.499690 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:49.499642    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" podStartSLOduration=2.858283419 podStartE2EDuration="28.499630139s" totalImagesPullingTime="21.837186068s" totalInitContainerRuntime="3.804160652s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:49.499035607 +0000 UTC m=+993.959953339" watchObservedRunningTime="2026-09-17 23:27:49.499630139 +0000 UTC m=+993.960547870"
Sep 17 23:27:50.484635 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:50.484608    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:27:56.493983 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:27:56.493958    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:28:26.495771 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:28:26.495733    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 23:28:36.494507 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:28:36.494472    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 23:28:46.497812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:28:46.497765    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 23:28:56.494785 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:28:56.494745    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.28:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.28:8080: connect: connection refused"
Sep 17 23:29:07.028464 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:07.028428    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:29:11.973886 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:11.973847    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"]
Sep 17 23:29:11.974307 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:11.974107    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kserve-container" containerID="cri-o://ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e" gracePeriod=30
Sep 17 23:29:11.974307 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:11.974141    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kube-rbac-proxy" containerID="cri-o://bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c" gracePeriod=30
Sep 17 23:29:12.040694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:12.040672    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:12.829303 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:12.829263    2571 generic.go:356] "Generic (PLEG): container finished" podID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerID="bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c" exitCode=2
Sep 17 23:29:12.829486 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:12.829347    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" event={"ID":"9805ee7e-7980-4c97-b1f3-7d09308c5bce","Type":"ContainerDied","Data":"bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c"}
Sep 17 23:29:16.317435 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.317410    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:29:16.438230 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.438169    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls\") pod \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") "
Sep 17 23:29:16.438230 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.438202    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-cabundle-cert\") pod \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") "
Sep 17 23:29:16.438420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.438232    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-rbac-proxy-sar-config\") pod \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") "
Sep 17 23:29:16.438420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.438315    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjm78\" (UniqueName: \"kubernetes.io/projected/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-api-access-wjm78\") pod \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") "
Sep 17 23:29:16.438420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.438365    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kserve-provision-location\") pod \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\" (UID: \"9805ee7e-7980-4c97-b1f3-7d09308c5bce\") "
Sep 17 23:29:16.438628 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.438594    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9805ee7e-7980-4c97-b1f3-7d09308c5bce" (UID: "9805ee7e-7980-4c97-b1f3-7d09308c5bce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:29:16.438745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.438625    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9805ee7e-7980-4c97-b1f3-7d09308c5bce" (UID: "9805ee7e-7980-4c97-b1f3-7d09308c5bce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:29:16.438811 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.438783    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9805ee7e-7980-4c97-b1f3-7d09308c5bce" (UID: "9805ee7e-7980-4c97-b1f3-7d09308c5bce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:16.440394 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.440375    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9805ee7e-7980-4c97-b1f3-7d09308c5bce" (UID: "9805ee7e-7980-4c97-b1f3-7d09308c5bce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:29:16.440394 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.440379    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-api-access-wjm78" (OuterVolumeSpecName: "kube-api-access-wjm78") pod "9805ee7e-7980-4c97-b1f3-7d09308c5bce" (UID: "9805ee7e-7980-4c97-b1f3-7d09308c5bce"). InnerVolumeSpecName "kube-api-access-wjm78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:29:16.539626 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.539600    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjm78\" (UniqueName: \"kubernetes.io/projected/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-api-access-wjm78\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.539626 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.539624    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.539763 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.539638    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9805ee7e-7980-4c97-b1f3-7d09308c5bce-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.539763 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.539651    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.539763 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.539665    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9805ee7e-7980-4c97-b1f3-7d09308c5bce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:29:16.847821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.847779    2571 generic.go:356] "Generic (PLEG): container finished" podID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerID="ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e" exitCode=0
Sep 17 23:29:16.847941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.847835    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" event={"ID":"9805ee7e-7980-4c97-b1f3-7d09308c5bce","Type":"ContainerDied","Data":"ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e"}
Sep 17 23:29:16.847941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.847856    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"
Sep 17 23:29:16.847941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.847894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4" event={"ID":"9805ee7e-7980-4c97-b1f3-7d09308c5bce","Type":"ContainerDied","Data":"dcacc9b368a0df64421e698573f5ff6f172a798cb97a3c8259f65a6e35bdc565"}
Sep 17 23:29:16.847941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.847924    2571 scope.go:117] "RemoveContainer" containerID="bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c"
Sep 17 23:29:16.857437 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.857420    2571 scope.go:117] "RemoveContainer" containerID="ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e"
Sep 17 23:29:16.865334 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.865299    2571 scope.go:117] "RemoveContainer" containerID="88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77"
Sep 17 23:29:16.872219 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.872194    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"]
Sep 17 23:29:16.873552 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.873538    2571 scope.go:117] "RemoveContainer" containerID="bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c"
Sep 17 23:29:16.873835 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:29:16.873815    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c\": container with ID starting with bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c not found: ID does not exist" containerID="bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c"
Sep 17 23:29:16.873921 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.873848    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c"} err="failed to get container status \"bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c\": rpc error: code = NotFound desc = could not find container \"bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c\": container with ID starting with bd35ab7ee2f3b812ef534c11291e52a966aece0f8e7b19b200de026f55efca9c not found: ID does not exist"
Sep 17 23:29:16.873921 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.873873    2571 scope.go:117] "RemoveContainer" containerID="ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e"
Sep 17 23:29:16.874141 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:29:16.874123    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e\": container with ID starting with ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e not found: ID does not exist" containerID="ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e"
Sep 17 23:29:16.874184 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.874148    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e"} err="failed to get container status \"ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e\": rpc error: code = NotFound desc = could not find container \"ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e\": container with ID starting with ed5f863a55c1df8f02ea597585d983cb568a938524e2b6ca2b27f5443b41155e not found: ID does not exist"
Sep 17 23:29:16.874184 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.874170    2571 scope.go:117] "RemoveContainer" containerID="88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77"
Sep 17 23:29:16.874421 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:29:16.874405    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77\": container with ID starting with 88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77 not found: ID does not exist" containerID="88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77"
Sep 17 23:29:16.874476 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.874426    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77"} err="failed to get container status \"88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77\": rpc error: code = NotFound desc = could not find container \"88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77\": container with ID starting with 88c898f3375cdd3b5f0eb7b836d9306a42a1810e8b4e06dd1a81e9306eb16f77 not found: ID does not exist"
Sep 17 23:29:16.878368 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:16.878348    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-2v5s4"]
Sep 17 23:29:17.060987 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.060958    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:17.107058 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.106990    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"]
Sep 17 23:29:17.107374 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107317    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kserve-container"
Sep 17 23:29:17.107454 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107379    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kserve-container"
Sep 17 23:29:17.107454 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107401    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy"
Sep 17 23:29:17.107454 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107409    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy"
Sep 17 23:29:17.107454 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107423    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container"
Sep 17 23:29:17.107454 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107431    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container"
Sep 17 23:29:17.107454 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107449    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="storage-initializer"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107459    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="storage-initializer"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107472    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107481    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107496    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kube-rbac-proxy"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107504    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kube-rbac-proxy"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107516    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="storage-initializer"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107524    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="storage-initializer"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107598    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kube-rbac-proxy"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107612    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kube-rbac-proxy"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107621    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="agent"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107629    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" containerName="kserve-container"
Sep 17 23:29:17.107765 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.107638    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="772765a5-3682-4b88-adcf-ac4a5527e8d4" containerName="kserve-container"
Sep 17 23:29:17.112154 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.112136    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.114794 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.114767    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 23:29:17.114913 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.114806    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:29:17.115051 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.115036    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:29:17.115093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.115041    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:29:17.115170 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.115142    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:29:17.115333 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.115306    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:29:17.115398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.115380    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 23:29:17.120247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.120225    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"]
Sep 17 23:29:17.246025 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.246000    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.246152 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.246034    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.246152 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.246058    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4h6s\" (UniqueName: \"kubernetes.io/projected/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-api-access-c4h6s\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.246152 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.246087    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.246152 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.246108    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.347364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.347313    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.347739 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.347369    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.347739 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.347416    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4h6s\" (UniqueName: \"kubernetes.io/projected/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-api-access-c4h6s\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.347739 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.347444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.347739 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.347470    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.347874 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.347804    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.348066 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.348035    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.348183 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.348106    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.350147 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.350126    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.356381 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.356362    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4h6s\" (UniqueName: \"kubernetes.io/projected/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-api-access-c4h6s\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.376571 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.376513    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:17.423249 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.423227    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:17.696097 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.696071    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"]
Sep 17 23:29:17.698211 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:29:17.698185    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf613eb_bfd3_41e5_b772_fc53ff9a8894.slice/crio-fd80ba7efd84597b41a0d8b85f33ab687cc4eff5c3efe45e05ce280589edff51 WatchSource:0}: Error finding container fd80ba7efd84597b41a0d8b85f33ab687cc4eff5c3efe45e05ce280589edff51: Status 404 returned error can't find the container with id fd80ba7efd84597b41a0d8b85f33ab687cc4eff5c3efe45e05ce280589edff51
Sep 17 23:29:17.854792 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.854763    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" event={"ID":"cdf613eb-bfd3-41e5-b772-fc53ff9a8894","Type":"ContainerStarted","Data":"0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c"}
Sep 17 23:29:17.854912 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:17.854816    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" event={"ID":"cdf613eb-bfd3-41e5-b772-fc53ff9a8894","Type":"ContainerStarted","Data":"fd80ba7efd84597b41a0d8b85f33ab687cc4eff5c3efe45e05ce280589edff51"}
Sep 17 23:29:18.063141 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:18.063065    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9805ee7e-7980-4c97-b1f3-7d09308c5bce" path="/var/lib/kubelet/pods/9805ee7e-7980-4c97-b1f3-7d09308c5bce/volumes"
Sep 17 23:29:21.872927 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:21.872898    2571 generic.go:356] "Generic (PLEG): container finished" podID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerID="0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c" exitCode=0
Sep 17 23:29:21.873266 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:21.872975    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" event={"ID":"cdf613eb-bfd3-41e5-b772-fc53ff9a8894","Type":"ContainerDied","Data":"0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c"}
Sep 17 23:29:22.879423 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:22.879383    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" event={"ID":"cdf613eb-bfd3-41e5-b772-fc53ff9a8894","Type":"ContainerStarted","Data":"8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b"}
Sep 17 23:29:22.879824 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:22.879444    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" event={"ID":"cdf613eb-bfd3-41e5-b772-fc53ff9a8894","Type":"ContainerStarted","Data":"2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804"}
Sep 17 23:29:22.879824 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:22.879664    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:22.899630 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:22.899583    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podStartSLOduration=1.985170791 podStartE2EDuration="5.899571359s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.914400568s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:29:22.898259747 +0000 UTC m=+1087.359177480" watchObservedRunningTime="2026-09-17 23:29:22.899571359 +0000 UTC m=+1087.360489089"
Sep 17 23:29:23.884738 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:23.884705    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:29.893117 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:29.893083    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:29:49.239151 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:49.239114    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:54.393909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:54.393871    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:54.494978 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:54.494946    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:29:59.894712 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:29:59.894675    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 23:30:09.894138 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:09.894097    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 23:30:19.894515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:19.894451    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 23:30:29.894275 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:29.894234    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 23:30:40.429371 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:40.429272    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:30:47.167220 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:47.167186    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"]
Sep 17 23:30:47.167701 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:47.167437    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container" containerID="cri-o://2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804" gracePeriod=30
Sep 17 23:30:47.167701 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:47.167503    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kube-rbac-proxy" containerID="cri-o://8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b" gracePeriod=30
Sep 17 23:30:47.194044 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:47.194013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:48.232140 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:48.232090    2571 generic.go:356] "Generic (PLEG): container finished" podID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerID="8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b" exitCode=2
Sep 17 23:30:48.232564 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:48.232159    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" event={"ID":"cdf613eb-bfd3-41e5-b772-fc53ff9a8894","Type":"ContainerDied","Data":"8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b"}
Sep 17 23:30:49.888873 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:49.888824    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Sep 17 23:30:49.893728 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:49.893694    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.29:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.29:8080: connect: connection refused"
Sep 17 23:30:51.508114 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.508089    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:30:51.612720 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.612690    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-cabundle-cert\") pod \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") "
Sep 17 23:30:51.612846 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.612727    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-rbac-proxy-sar-config\") pod \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") "
Sep 17 23:30:51.612846 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.612758    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4h6s\" (UniqueName: \"kubernetes.io/projected/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-api-access-c4h6s\") pod \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") "
Sep 17 23:30:51.612846 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.612785    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-proxy-tls\") pod \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") "
Sep 17 23:30:51.613019 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.612846    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kserve-provision-location\") pod \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\" (UID: \"cdf613eb-bfd3-41e5-b772-fc53ff9a8894\") "
Sep 17 23:30:51.613168 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.613148    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cdf613eb-bfd3-41e5-b772-fc53ff9a8894" (UID: "cdf613eb-bfd3-41e5-b772-fc53ff9a8894"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:51.613226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.613167    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cdf613eb-bfd3-41e5-b772-fc53ff9a8894" (UID: "cdf613eb-bfd3-41e5-b772-fc53ff9a8894"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:30:51.613226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.613191    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdf613eb-bfd3-41e5-b772-fc53ff9a8894" (UID: "cdf613eb-bfd3-41e5-b772-fc53ff9a8894"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:30:51.614994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.614978    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cdf613eb-bfd3-41e5-b772-fc53ff9a8894" (UID: "cdf613eb-bfd3-41e5-b772-fc53ff9a8894"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:30:51.615135 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.615115    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-api-access-c4h6s" (OuterVolumeSpecName: "kube-api-access-c4h6s") pod "cdf613eb-bfd3-41e5-b772-fc53ff9a8894" (UID: "cdf613eb-bfd3-41e5-b772-fc53ff9a8894"). InnerVolumeSpecName "kube-api-access-c4h6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:30:51.713375 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.713350    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:30:51.713375 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.713369    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:30:51.713375 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.713379    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:30:51.713531 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.713387    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4h6s\" (UniqueName: \"kubernetes.io/projected/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-kube-api-access-c4h6s\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:30:51.713531 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:51.713395    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cdf613eb-bfd3-41e5-b772-fc53ff9a8894-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:30:52.251912 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.251883    2571 generic.go:356] "Generic (PLEG): container finished" podID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerID="2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804" exitCode=0
Sep 17 23:30:52.252062 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.251953    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"
Sep 17 23:30:52.252062 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.251958    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" event={"ID":"cdf613eb-bfd3-41e5-b772-fc53ff9a8894","Type":"ContainerDied","Data":"2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804"}
Sep 17 23:30:52.252062 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.252020    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z" event={"ID":"cdf613eb-bfd3-41e5-b772-fc53ff9a8894","Type":"ContainerDied","Data":"fd80ba7efd84597b41a0d8b85f33ab687cc4eff5c3efe45e05ce280589edff51"}
Sep 17 23:30:52.252062 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.252050    2571 scope.go:117] "RemoveContainer" containerID="8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b"
Sep 17 23:30:52.260667 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.260614    2571 scope.go:117] "RemoveContainer" containerID="2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804"
Sep 17 23:30:52.268227 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.268210    2571 scope.go:117] "RemoveContainer" containerID="0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c"
Sep 17 23:30:52.273747 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.273727    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"]
Sep 17 23:30:52.276478 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.276430    2571 scope.go:117] "RemoveContainer" containerID="8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b"
Sep 17 23:30:52.276750 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:30:52.276732    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b\": container with ID starting with 8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b not found: ID does not exist" containerID="8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b"
Sep 17 23:30:52.276823 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.276763    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b"} err="failed to get container status \"8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b\": rpc error: code = NotFound desc = could not find container \"8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b\": container with ID starting with 8a06911439ad3cfca74eed36ac6bd372bd50b3b6a4fff5d479bf7031aaff1d7b not found: ID does not exist"
Sep 17 23:30:52.276823 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.276788    2571 scope.go:117] "RemoveContainer" containerID="2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804"
Sep 17 23:30:52.277068 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:30:52.277047    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804\": container with ID starting with 2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804 not found: ID does not exist" containerID="2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804"
Sep 17 23:30:52.277121 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.277078    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804"} err="failed to get container status \"2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804\": rpc error: code = NotFound desc = could not find container \"2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804\": container with ID starting with 2e0263f5d15fa7263746a4715353e8fa90458368412e75b8fc9221e56cae8804 not found: ID does not exist"
Sep 17 23:30:52.277121 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.277100    2571 scope.go:117] "RemoveContainer" containerID="0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c"
Sep 17 23:30:52.277446 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:30:52.277425    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c\": container with ID starting with 0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c not found: ID does not exist" containerID="0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c"
Sep 17 23:30:52.277524 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.277452    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c"} err="failed to get container status \"0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c\": rpc error: code = NotFound desc = could not find container \"0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c\": container with ID starting with 0ee9b9306dfb99a6e9ecd39c8a20cf75839f283308cfc4aef1398020fa22b09c not found: ID does not exist"
Sep 17 23:30:52.277806 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.277788    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-2jr8z"]
Sep 17 23:30:52.376128 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.376104    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.379584 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379562    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"]
Sep 17 23:30:52.379858 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379846    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container"
Sep 17 23:30:52.379899 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379860    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container"
Sep 17 23:30:52.379899 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379868    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kube-rbac-proxy"
Sep 17 23:30:52.379899 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379874    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kube-rbac-proxy"
Sep 17 23:30:52.379899 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379884    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="storage-initializer"
Sep 17 23:30:52.379899 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379890    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="storage-initializer"
Sep 17 23:30:52.380053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379933    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kserve-container"
Sep 17 23:30:52.380053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.379944    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" containerName="kube-rbac-proxy"
Sep 17 23:30:52.384308 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.384293    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.386830 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.386807    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 23:30:52.387106 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.387082    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:30:52.387175 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.387123    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:30:52.387175 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.387164    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 23:30:52.387345 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.387171    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:30:52.387345 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.387233    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:30:52.387422 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.387404    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:30:52.392979 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.392960    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"]
Sep 17 23:30:52.419402 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.419377    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfq2d\" (UniqueName: \"kubernetes.io/projected/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-api-access-hfq2d\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.419509 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.419404    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.419509 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.419491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691a547d-2b59-4c16-bafe-a8fec56f632e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.419624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.419525    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691a547d-2b59-4c16-bafe-a8fec56f632e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.419624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.419555    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.520711 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.520687    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfq2d\" (UniqueName: \"kubernetes.io/projected/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-api-access-hfq2d\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.521054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.520714    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.521054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.520745    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691a547d-2b59-4c16-bafe-a8fec56f632e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.521054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.520767    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691a547d-2b59-4c16-bafe-a8fec56f632e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.521054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.520788    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.521192 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.521163    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691a547d-2b59-4c16-bafe-a8fec56f632e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.521414 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.521395    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.521467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.521429    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.523253 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.523236    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691a547d-2b59-4c16-bafe-a8fec56f632e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.525259 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.525237    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:30:52.529746 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.529725    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfq2d\" (UniqueName: \"kubernetes.io/projected/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-api-access-hfq2d\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.694390 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.694341    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:52.970126 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:52.970022    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"]
Sep 17 23:30:52.972203 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:30:52.972177    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691a547d_2b59_4c16_bafe_a8fec56f632e.slice/crio-bd2d98ee31490713cf9389544d4bf341b9e3c71af4ade7f6b3dd576d4652de03 WatchSource:0}: Error finding container bd2d98ee31490713cf9389544d4bf341b9e3c71af4ade7f6b3dd576d4652de03: Status 404 returned error can't find the container with id bd2d98ee31490713cf9389544d4bf341b9e3c71af4ade7f6b3dd576d4652de03
Sep 17 23:30:53.258187 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:53.258119    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" event={"ID":"691a547d-2b59-4c16-bafe-a8fec56f632e","Type":"ContainerStarted","Data":"f9e7cbc5d4997cfbe33cd1ed194aebb714abfc3ccc7cbac7e8b1f6ea75635b80"}
Sep 17 23:30:53.258339 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:53.258198    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" event={"ID":"691a547d-2b59-4c16-bafe-a8fec56f632e","Type":"ContainerStarted","Data":"bd2d98ee31490713cf9389544d4bf341b9e3c71af4ade7f6b3dd576d4652de03"}
Sep 17 23:30:54.058960 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:54.058921    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdf613eb-bfd3-41e5-b772-fc53ff9a8894" path="/var/lib/kubelet/pods/cdf613eb-bfd3-41e5-b772-fc53ff9a8894/volumes"
Sep 17 23:30:57.279876 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:57.279840    2571 generic.go:356] "Generic (PLEG): container finished" podID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerID="f9e7cbc5d4997cfbe33cd1ed194aebb714abfc3ccc7cbac7e8b1f6ea75635b80" exitCode=0
Sep 17 23:30:57.280239 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:57.279884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" event={"ID":"691a547d-2b59-4c16-bafe-a8fec56f632e","Type":"ContainerDied","Data":"f9e7cbc5d4997cfbe33cd1ed194aebb714abfc3ccc7cbac7e8b1f6ea75635b80"}
Sep 17 23:30:58.285909 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:58.285877    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" event={"ID":"691a547d-2b59-4c16-bafe-a8fec56f632e","Type":"ContainerStarted","Data":"94b6231ef1480ec726c4b23a91374c2e58f0d34bc9210c39eb82ebd7f7f80aa3"}
Sep 17 23:30:58.286286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:58.285932    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" event={"ID":"691a547d-2b59-4c16-bafe-a8fec56f632e","Type":"ContainerStarted","Data":"4ce0e307caf5cb00c2e99e8861cc6c38a212f702da03a94fc1833e7807c860d2"}
Sep 17 23:30:58.286286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:58.286189    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:58.286286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:58.286260    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:30:58.310273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:30:58.310228    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podStartSLOduration=2.662894936 podStartE2EDuration="6.310215086s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.64732015s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:58.308433375 +0000 UTC m=+1182.769351106" watchObservedRunningTime="2026-09-17 23:30:58.310215086 +0000 UTC m=+1182.771132817"
Sep 17 23:31:03.609608 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:03.609575    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:04.296541 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:04.296516    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:31:16.053361 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:16.053309    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:31:16.053755 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:16.053381    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:31:16.085368 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:16.085344    2571 scope.go:117] "RemoveContainer" containerID="a077639d67e7f7725113888c92e731988a3c30793d7f7596465a1092dc247960"
Sep 17 23:31:16.092929 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:16.092912    2571 scope.go:117] "RemoveContainer" containerID="cf549fd6f8b959381e45bfe2b534f149ef183ef00986470ca13e5b1aa1d6ddcb"
Sep 17 23:31:34.298139 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:34.298098    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 23:31:37.311470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:37.311431    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:40.571103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:40.571047    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:44.297313 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:44.297279    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 23:31:45.718620 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:45.718585    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:46.115047 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:46.115009    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:31:54.297104 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:31:54.297065    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 23:32:04.297439 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:04.297395    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 23:32:14.736613 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:14.736536    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:32:19.413776 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:19.413740    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:22.519989 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:22.519955    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"]
Sep 17 23:32:22.520384 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:22.520180    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container" containerID="cri-o://4ce0e307caf5cb00c2e99e8861cc6c38a212f702da03a94fc1833e7807c860d2" gracePeriod=30
Sep 17 23:32:22.520384 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:22.520219    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kube-rbac-proxy" containerID="cri-o://94b6231ef1480ec726c4b23a91374c2e58f0d34bc9210c39eb82ebd7f7f80aa3" gracePeriod=30
Sep 17 23:32:22.531226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:22.531199    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:23.623283 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:23.623254    2571 generic.go:356] "Generic (PLEG): container finished" podID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerID="94b6231ef1480ec726c4b23a91374c2e58f0d34bc9210c39eb82ebd7f7f80aa3" exitCode=2
Sep 17 23:32:23.623652 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:23.623311    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" event={"ID":"691a547d-2b59-4c16-bafe-a8fec56f632e","Type":"ContainerDied","Data":"94b6231ef1480ec726c4b23a91374c2e58f0d34bc9210c39eb82ebd7f7f80aa3"}
Sep 17 23:32:24.290578 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:24.290545    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Sep 17 23:32:24.297013 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:24.296989    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.30:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.30:8080: connect: connection refused"
Sep 17 23:32:27.642076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:27.642047    2571 generic.go:356] "Generic (PLEG): container finished" podID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerID="4ce0e307caf5cb00c2e99e8861cc6c38a212f702da03a94fc1833e7807c860d2" exitCode=0
Sep 17 23:32:27.642398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:27.642120    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" event={"ID":"691a547d-2b59-4c16-bafe-a8fec56f632e","Type":"ContainerDied","Data":"4ce0e307caf5cb00c2e99e8861cc6c38a212f702da03a94fc1833e7807c860d2"}
Sep 17 23:32:27.655672 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:27.655637    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:27.884272 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:27.884250    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:28.059544 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.059520    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:32:28.104033 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104011    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-cabundle-cert\") pod \"691a547d-2b59-4c16-bafe-a8fec56f632e\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") "
Sep 17 23:32:28.104170 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104049    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691a547d-2b59-4c16-bafe-a8fec56f632e-proxy-tls\") pod \"691a547d-2b59-4c16-bafe-a8fec56f632e\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") "
Sep 17 23:32:28.104170 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104111    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfq2d\" (UniqueName: \"kubernetes.io/projected/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-api-access-hfq2d\") pod \"691a547d-2b59-4c16-bafe-a8fec56f632e\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") "
Sep 17 23:32:28.104170 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104151    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-rbac-proxy-sar-config\") pod \"691a547d-2b59-4c16-bafe-a8fec56f632e\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") "
Sep 17 23:32:28.104417 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104178    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691a547d-2b59-4c16-bafe-a8fec56f632e-kserve-provision-location\") pod \"691a547d-2b59-4c16-bafe-a8fec56f632e\" (UID: \"691a547d-2b59-4c16-bafe-a8fec56f632e\") "
Sep 17 23:32:28.104417 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104385    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "691a547d-2b59-4c16-bafe-a8fec56f632e" (UID: "691a547d-2b59-4c16-bafe-a8fec56f632e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:28.104539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104467    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:32:28.104539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104499    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "691a547d-2b59-4c16-bafe-a8fec56f632e" (UID: "691a547d-2b59-4c16-bafe-a8fec56f632e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:28.104636 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.104555    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/691a547d-2b59-4c16-bafe-a8fec56f632e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "691a547d-2b59-4c16-bafe-a8fec56f632e" (UID: "691a547d-2b59-4c16-bafe-a8fec56f632e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:28.106291 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.106265    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/691a547d-2b59-4c16-bafe-a8fec56f632e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "691a547d-2b59-4c16-bafe-a8fec56f632e" (UID: "691a547d-2b59-4c16-bafe-a8fec56f632e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:28.106413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.106298    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-api-access-hfq2d" (OuterVolumeSpecName: "kube-api-access-hfq2d") pod "691a547d-2b59-4c16-bafe-a8fec56f632e" (UID: "691a547d-2b59-4c16-bafe-a8fec56f632e"). InnerVolumeSpecName "kube-api-access-hfq2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:28.204960 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.204915    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/691a547d-2b59-4c16-bafe-a8fec56f632e-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:32:28.204960 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.204934    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfq2d\" (UniqueName: \"kubernetes.io/projected/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-api-access-hfq2d\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:32:28.204960 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.204944    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/691a547d-2b59-4c16-bafe-a8fec56f632e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:32:28.204960 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.204954    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/691a547d-2b59-4c16-bafe-a8fec56f632e-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:32:28.648031 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.648010    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"
Sep 17 23:32:28.648424 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.647997    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4" event={"ID":"691a547d-2b59-4c16-bafe-a8fec56f632e","Type":"ContainerDied","Data":"bd2d98ee31490713cf9389544d4bf341b9e3c71af4ade7f6b3dd576d4652de03"}
Sep 17 23:32:28.648424 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.648148    2571 scope.go:117] "RemoveContainer" containerID="94b6231ef1480ec726c4b23a91374c2e58f0d34bc9210c39eb82ebd7f7f80aa3"
Sep 17 23:32:28.656644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.656627    2571 scope.go:117] "RemoveContainer" containerID="4ce0e307caf5cb00c2e99e8861cc6c38a212f702da03a94fc1833e7807c860d2"
Sep 17 23:32:28.664021 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.664006    2571 scope.go:117] "RemoveContainer" containerID="f9e7cbc5d4997cfbe33cd1ed194aebb714abfc3ccc7cbac7e8b1f6ea75635b80"
Sep 17 23:32:28.670678 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.670657    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"]
Sep 17 23:32:28.676198 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:28.676175    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-gv5t4"]
Sep 17 23:32:29.694615 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:29.694585    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:32:30.057642 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:30.057574    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" path="/var/lib/kubelet/pods/691a547d-2b59-4c16-bafe-a8fec56f632e/volumes"
Sep 17 23:32:45.215639 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215597    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-c785f49b"]
Sep 17 23:32:45.216038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215870    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="storage-initializer"
Sep 17 23:32:45.216038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215881    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="storage-initializer"
Sep 17 23:32:45.216038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215895    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kube-rbac-proxy"
Sep 17 23:32:45.216038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215900    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kube-rbac-proxy"
Sep 17 23:32:45.216038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215911    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container"
Sep 17 23:32:45.216038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215917    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container"
Sep 17 23:32:45.216038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215969    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kube-rbac-proxy"
Sep 17 23:32:45.216038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.215978    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="691a547d-2b59-4c16-bafe-a8fec56f632e" containerName="kserve-container"
Sep 17 23:32:45.218707 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.218692    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-c785f49b"
Sep 17 23:32:45.221406 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.221381    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:32:45.221539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.221381    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:32:45.222537 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.222521    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8hff\""
Sep 17 23:32:45.226075 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.226058    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-c785f49b"]
Sep 17 23:32:45.321291 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.321270    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gvc4\" (UniqueName: \"kubernetes.io/projected/7d86be86-f9a9-434e-8358-00dee112e84a-kube-api-access-6gvc4\") pod \"curl-metrics-c785f49b\" (UID: \"7d86be86-f9a9-434e-8358-00dee112e84a\") " pod="kserve/curl-metrics-c785f49b"
Sep 17 23:32:45.421901 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.421878    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gvc4\" (UniqueName: \"kubernetes.io/projected/7d86be86-f9a9-434e-8358-00dee112e84a-kube-api-access-6gvc4\") pod \"curl-metrics-c785f49b\" (UID: \"7d86be86-f9a9-434e-8358-00dee112e84a\") " pod="kserve/curl-metrics-c785f49b"
Sep 17 23:32:45.431037 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.431017    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gvc4\" (UniqueName: \"kubernetes.io/projected/7d86be86-f9a9-434e-8358-00dee112e84a-kube-api-access-6gvc4\") pod \"curl-metrics-c785f49b\" (UID: \"7d86be86-f9a9-434e-8358-00dee112e84a\") " pod="kserve/curl-metrics-c785f49b"
Sep 17 23:32:45.537653 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.537608    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-c785f49b"
Sep 17 23:32:45.805942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.805919    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-c785f49b"]
Sep 17 23:32:45.808288 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:32:45.808252    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d86be86_f9a9_434e_8358_00dee112e84a.slice/crio-5e6e6bfcf9b038e418e6f25c9599ee2fbcf9bcf45127b6e6be3ee84722746577 WatchSource:0}: Error finding container 5e6e6bfcf9b038e418e6f25c9599ee2fbcf9bcf45127b6e6be3ee84722746577: Status 404 returned error can't find the container with id 5e6e6bfcf9b038e418e6f25c9599ee2fbcf9bcf45127b6e6be3ee84722746577
Sep 17 23:32:45.810243 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:45.810224    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:32:46.728835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:46.728796    2571 generic.go:356] "Generic (PLEG): container finished" podID="7d86be86-f9a9-434e-8358-00dee112e84a" containerID="4e4a502aaac0026736d5137f60758d4cc77e0748ec1f12d9cece74856c89169e" exitCode=0
Sep 17 23:32:46.729153 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:46.728888    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-c785f49b" event={"ID":"7d86be86-f9a9-434e-8358-00dee112e84a","Type":"ContainerDied","Data":"4e4a502aaac0026736d5137f60758d4cc77e0748ec1f12d9cece74856c89169e"}
Sep 17 23:32:46.729153 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:46.728941    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-c785f49b" event={"ID":"7d86be86-f9a9-434e-8358-00dee112e84a","Type":"ContainerStarted","Data":"5e6e6bfcf9b038e418e6f25c9599ee2fbcf9bcf45127b6e6be3ee84722746577"}
Sep 17 23:32:47.856484 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:47.856461    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-c785f49b"
Sep 17 23:32:47.877442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:47.877416    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-c785f49b_7d86be86-f9a9-434e-8358-00dee112e84a/curl-metrics-c785f49b/0.log"
Sep 17 23:32:47.901528 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:47.901501    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-c785f49b"]
Sep 17 23:32:47.907505 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:47.907484    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-c785f49b"]
Sep 17 23:32:47.942428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:47.942405    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gvc4\" (UniqueName: \"kubernetes.io/projected/7d86be86-f9a9-434e-8358-00dee112e84a-kube-api-access-6gvc4\") pod \"7d86be86-f9a9-434e-8358-00dee112e84a\" (UID: \"7d86be86-f9a9-434e-8358-00dee112e84a\") "
Sep 17 23:32:47.944502 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:47.944480    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d86be86-f9a9-434e-8358-00dee112e84a-kube-api-access-6gvc4" (OuterVolumeSpecName: "kube-api-access-6gvc4") pod "7d86be86-f9a9-434e-8358-00dee112e84a" (UID: "7d86be86-f9a9-434e-8358-00dee112e84a"). InnerVolumeSpecName "kube-api-access-6gvc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:48.042991 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:48.042934    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gvc4\" (UniqueName: \"kubernetes.io/projected/7d86be86-f9a9-434e-8358-00dee112e84a-kube-api-access-6gvc4\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:32:48.058158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:48.058135    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d86be86-f9a9-434e-8358-00dee112e84a" path="/var/lib/kubelet/pods/7d86be86-f9a9-434e-8358-00dee112e84a/volumes"
Sep 17 23:32:48.738662 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:48.738637    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-c785f49b"
Sep 17 23:32:48.738832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:48.738636    2571 scope.go:117] "RemoveContainer" containerID="4e4a502aaac0026736d5137f60758d4cc77e0748ec1f12d9cece74856c89169e"
Sep 17 23:32:53.300185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.300143    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-3acc3c36"]
Sep 17 23:32:53.300587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.300488    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d86be86-f9a9-434e-8358-00dee112e84a" containerName="curl-metrics-c785f49b"
Sep 17 23:32:53.300587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.300502    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d86be86-f9a9-434e-8358-00dee112e84a" containerName="curl-metrics-c785f49b"
Sep 17 23:32:53.300587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.300562    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d86be86-f9a9-434e-8358-00dee112e84a" containerName="curl-metrics-c785f49b"
Sep 17 23:32:53.304841 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.304823    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-3acc3c36"
Sep 17 23:32:53.307835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.307811    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:32:53.308182 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.308160    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:32:53.309122 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.309105    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-q8hff\""
Sep 17 23:32:53.315648 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.315620    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-3acc3c36"]
Sep 17 23:32:53.381404 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.381372    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbq5g\" (UniqueName: \"kubernetes.io/projected/9e63cb30-9118-4a9d-84e5-08bdbe175ce7-kube-api-access-bbq5g\") pod \"curl-metrics-3acc3c36\" (UID: \"9e63cb30-9118-4a9d-84e5-08bdbe175ce7\") " pod="kserve/curl-metrics-3acc3c36"
Sep 17 23:32:53.482497 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.482457    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbq5g\" (UniqueName: \"kubernetes.io/projected/9e63cb30-9118-4a9d-84e5-08bdbe175ce7-kube-api-access-bbq5g\") pod \"curl-metrics-3acc3c36\" (UID: \"9e63cb30-9118-4a9d-84e5-08bdbe175ce7\") " pod="kserve/curl-metrics-3acc3c36"
Sep 17 23:32:53.498725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.498693    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbq5g\" (UniqueName: \"kubernetes.io/projected/9e63cb30-9118-4a9d-84e5-08bdbe175ce7-kube-api-access-bbq5g\") pod \"curl-metrics-3acc3c36\" (UID: \"9e63cb30-9118-4a9d-84e5-08bdbe175ce7\") " pod="kserve/curl-metrics-3acc3c36"
Sep 17 23:32:53.633893 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.633870    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-3acc3c36"
Sep 17 23:32:53.909156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:53.909131    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-3acc3c36"]
Sep 17 23:32:53.911722 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:32:53.911695    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e63cb30_9118_4a9d_84e5_08bdbe175ce7.slice/crio-be6d47d0b0bff080b767befbcf6d0a0de4a6c90187a872a993fc2f555787aa50 WatchSource:0}: Error finding container be6d47d0b0bff080b767befbcf6d0a0de4a6c90187a872a993fc2f555787aa50: Status 404 returned error can't find the container with id be6d47d0b0bff080b767befbcf6d0a0de4a6c90187a872a993fc2f555787aa50
Sep 17 23:32:54.766536 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:54.766447    2571 generic.go:356] "Generic (PLEG): container finished" podID="9e63cb30-9118-4a9d-84e5-08bdbe175ce7" containerID="0f3324bab6f2f1ab8c087500e7d99ffb9797862ebc6d3fbbd853ffd138c4f263" exitCode=0
Sep 17 23:32:54.766536 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:54.766515    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-3acc3c36" event={"ID":"9e63cb30-9118-4a9d-84e5-08bdbe175ce7","Type":"ContainerDied","Data":"0f3324bab6f2f1ab8c087500e7d99ffb9797862ebc6d3fbbd853ffd138c4f263"}
Sep 17 23:32:54.766955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:54.766550    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-3acc3c36" event={"ID":"9e63cb30-9118-4a9d-84e5-08bdbe175ce7","Type":"ContainerStarted","Data":"be6d47d0b0bff080b767befbcf6d0a0de4a6c90187a872a993fc2f555787aa50"}
Sep 17 23:32:55.890105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:55.890084    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-3acc3c36"
Sep 17 23:32:55.912622 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:55.912603    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-3acc3c36_9e63cb30-9118-4a9d-84e5-08bdbe175ce7/curl-metrics-3acc3c36/0.log"
Sep 17 23:32:55.938853 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:55.938826    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-3acc3c36"]
Sep 17 23:32:55.951896 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:55.951871    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-3acc3c36"]
Sep 17 23:32:56.012930 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:56.012909    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbq5g\" (UniqueName: \"kubernetes.io/projected/9e63cb30-9118-4a9d-84e5-08bdbe175ce7-kube-api-access-bbq5g\") pod \"9e63cb30-9118-4a9d-84e5-08bdbe175ce7\" (UID: \"9e63cb30-9118-4a9d-84e5-08bdbe175ce7\") "
Sep 17 23:32:56.014971 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:56.014952    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e63cb30-9118-4a9d-84e5-08bdbe175ce7-kube-api-access-bbq5g" (OuterVolumeSpecName: "kube-api-access-bbq5g") pod "9e63cb30-9118-4a9d-84e5-08bdbe175ce7" (UID: "9e63cb30-9118-4a9d-84e5-08bdbe175ce7"). InnerVolumeSpecName "kube-api-access-bbq5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:56.058224 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:56.058155    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e63cb30-9118-4a9d-84e5-08bdbe175ce7" path="/var/lib/kubelet/pods/9e63cb30-9118-4a9d-84e5-08bdbe175ce7/volumes"
Sep 17 23:32:56.113572 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:56.113548    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbq5g\" (UniqueName: \"kubernetes.io/projected/9e63cb30-9118-4a9d-84e5-08bdbe175ce7-kube-api-access-bbq5g\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:32:56.776690 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:56.776662    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-3acc3c36"
Sep 17 23:32:56.776863 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:32:56.776712    2571 scope.go:117] "RemoveContainer" containerID="0f3324bab6f2f1ab8c087500e7d99ffb9797862ebc6d3fbbd853ffd138c4f263"
Sep 17 23:33:01.312258 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.312213    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:01.313400 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.313374    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"]
Sep 17 23:33:01.313751 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.313732    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e63cb30-9118-4a9d-84e5-08bdbe175ce7" containerName="curl-metrics-3acc3c36"
Sep 17 23:33:01.313827 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.313754    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e63cb30-9118-4a9d-84e5-08bdbe175ce7" containerName="curl-metrics-3acc3c36"
Sep 17 23:33:01.313884 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.313855    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e63cb30-9118-4a9d-84e5-08bdbe175ce7" containerName="curl-metrics-3acc3c36"
Sep 17 23:33:01.318265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.318245    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.320875 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.320853    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:33:01.321180 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.321152    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:33:01.321233 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.321152    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 23:33:01.322212 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.322192    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 23:33:01.322281 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.322193    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:33:01.322281 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.322195    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:33:01.322281 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.322197    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:33:01.324623 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.324598    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"]
Sep 17 23:33:01.445899 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.445873    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:33:01.452887 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.452864    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzsvl\" (UniqueName: \"kubernetes.io/projected/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-api-access-dzsvl\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.452970 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.452901    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kserve-provision-location\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.452970 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.452933    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-cabundle-cert\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.453046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.453019    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.453080 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.453051    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-proxy-tls\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.554338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.554305    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kserve-provision-location\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.554432 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.554383    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-cabundle-cert\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.554432 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.554425    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.554571 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.554459    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-proxy-tls\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.554571 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.554511    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzsvl\" (UniqueName: \"kubernetes.io/projected/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-api-access-dzsvl\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.554737 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.554717    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kserve-provision-location\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.555081 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.555061    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-cabundle-cert\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.555145 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.555065    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.557035 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.557019    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-proxy-tls\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.563882 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.563834    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzsvl\" (UniqueName: \"kubernetes.io/projected/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-api-access-dzsvl\") pod \"isvc-sklearn-predictor-799dc7b467-d9tpj\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.629820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.629800    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:01.904221 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:01.904122    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"]
Sep 17 23:33:01.906929 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:33:01.906901    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66411ed3_632a_4ec7_96f6_ae4eaaafd7d2.slice/crio-bbed856e864c29481968cf8510ae2a1eecba2ff80bea9b9b46d75ef9e049cdef WatchSource:0}: Error finding container bbed856e864c29481968cf8510ae2a1eecba2ff80bea9b9b46d75ef9e049cdef: Status 404 returned error can't find the container with id bbed856e864c29481968cf8510ae2a1eecba2ff80bea9b9b46d75ef9e049cdef
Sep 17 23:33:02.810167 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:02.810133    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" event={"ID":"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2","Type":"ContainerStarted","Data":"62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9"}
Sep 17 23:33:02.810638 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:02.810205    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" event={"ID":"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2","Type":"ContainerStarted","Data":"bbed856e864c29481968cf8510ae2a1eecba2ff80bea9b9b46d75ef9e049cdef"}
Sep 17 23:33:06.827550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:06.827513    2571 generic.go:356] "Generic (PLEG): container finished" podID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerID="62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9" exitCode=0
Sep 17 23:33:06.827916 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:06.827572    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" event={"ID":"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2","Type":"ContainerDied","Data":"62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9"}
Sep 17 23:33:07.834702 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:07.834665    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" event={"ID":"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2","Type":"ContainerStarted","Data":"dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb"}
Sep 17 23:33:07.835067 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:07.834757    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" event={"ID":"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2","Type":"ContainerStarted","Data":"357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3"}
Sep 17 23:33:07.835067 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:07.834951    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:07.855292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:07.855239    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podStartSLOduration=2.876436346 podStartE2EDuration="6.855226305s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.978789959s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:07.853031741 +0000 UTC m=+1312.313949473" watchObservedRunningTime="2026-09-17 23:33:07.855226305 +0000 UTC m=+1312.316144035"
Sep 17 23:33:08.839166 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:08.839136    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:08.840388 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:08.840359    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:33:09.842981 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:09.842944    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:33:14.846969 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:14.846945    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:33:14.847445 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:14.847417    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:33:24.848176 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:24.848138    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:33:34.847959 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:34.847922    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:33:44.847890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:44.847811    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:33:54.847823 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:33:54.847782    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:34:02.790210 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:02.790170    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:04.848477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:04.848046    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:34:07.894176 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:07.894137    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:08.267293 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:08.267212    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:14.847677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:14.847641    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:34:24.848313 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:24.848279    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:34:31.414546 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:31.414498    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"]
Sep 17 23:34:31.415125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:31.414863    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" containerID="cri-o://357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3" gracePeriod=30
Sep 17 23:34:31.415125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:31.414994    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kube-rbac-proxy" containerID="cri-o://dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb" gracePeriod=30
Sep 17 23:34:31.447954 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:31.447931    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:32.185187 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:32.185154    2571 generic.go:356] "Generic (PLEG): container finished" podID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerID="dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb" exitCode=2
Sep 17 23:34:32.185187 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:32.185190    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" event={"ID":"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2","Type":"ContainerDied","Data":"dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb"}
Sep 17 23:34:34.843623 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:34.843588    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 23:34:34.847934 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:34.847903    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 23:34:35.454314 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.454293    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:34:35.519373 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.519296    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzsvl\" (UniqueName: \"kubernetes.io/projected/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-api-access-dzsvl\") pod \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") "
Sep 17 23:34:35.519373 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.519343    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-cabundle-cert\") pod \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") "
Sep 17 23:34:35.519565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.519380    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-rbac-proxy-sar-config\") pod \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") "
Sep 17 23:34:35.519565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.519402    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kserve-provision-location\") pod \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") "
Sep 17 23:34:35.519565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.519420    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-proxy-tls\") pod \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\" (UID: \"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2\") "
Sep 17 23:34:35.519757 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.519726    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" (UID: "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:34:35.519811 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.519759    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" (UID: "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:34:35.519811 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.519767    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" (UID: "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:34:35.521462 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.521441    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-api-access-dzsvl" (OuterVolumeSpecName: "kube-api-access-dzsvl") pod "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" (UID: "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2"). InnerVolumeSpecName "kube-api-access-dzsvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:34:35.521562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.521522    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" (UID: "66411ed3-632a-4ec7-96f6-ae4eaaafd7d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:34:35.620779 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.620750    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzsvl\" (UniqueName: \"kubernetes.io/projected/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-api-access-dzsvl\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:34:35.620779 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.620776    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:34:35.620919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.620790    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:34:35.620919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.620802    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:34:35.620919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:35.620815    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:34:36.207715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.207681    2571 generic.go:356] "Generic (PLEG): container finished" podID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerID="357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3" exitCode=0
Sep 17 23:34:36.208139 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.207730    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" event={"ID":"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2","Type":"ContainerDied","Data":"357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3"}
Sep 17 23:34:36.208139 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.207748    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"
Sep 17 23:34:36.208139 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.207779    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj" event={"ID":"66411ed3-632a-4ec7-96f6-ae4eaaafd7d2","Type":"ContainerDied","Data":"bbed856e864c29481968cf8510ae2a1eecba2ff80bea9b9b46d75ef9e049cdef"}
Sep 17 23:34:36.208139 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.207808    2571 scope.go:117] "RemoveContainer" containerID="dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb"
Sep 17 23:34:36.216010 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.215994    2571 scope.go:117] "RemoveContainer" containerID="357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3"
Sep 17 23:34:36.223088 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.223069    2571 scope.go:117] "RemoveContainer" containerID="62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9"
Sep 17 23:34:36.229261 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.229240    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"]
Sep 17 23:34:36.230934 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.230899    2571 scope.go:117] "RemoveContainer" containerID="dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb"
Sep 17 23:34:36.236963 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:34:36.231163    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb\": container with ID starting with dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb not found: ID does not exist" containerID="dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb"
Sep 17 23:34:36.236963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.231501    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb"} err="failed to get container status \"dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb\": rpc error: code = NotFound desc = could not find container \"dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb\": container with ID starting with dfb9462e655ccdc6f5d8aa55c71356bc731343b7e959fbb2e77b9d4366adbefb not found: ID does not exist"
Sep 17 23:34:36.236963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.231694    2571 scope.go:117] "RemoveContainer" containerID="357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3"
Sep 17 23:34:36.236963 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:34:36.233139    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3\": container with ID starting with 357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3 not found: ID does not exist" containerID="357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3"
Sep 17 23:34:36.236963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.233177    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3"} err="failed to get container status \"357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3\": rpc error: code = NotFound desc = could not find container \"357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3\": container with ID starting with 357fb0e1d8a6a7407cb3375edb4fff07c2db5815754ede06379b6e91f01641e3 not found: ID does not exist"
Sep 17 23:34:36.236963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.233201    2571 scope.go:117] "RemoveContainer" containerID="62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9"
Sep 17 23:34:36.236963 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:34:36.233828    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9\": container with ID starting with 62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9 not found: ID does not exist" containerID="62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9"
Sep 17 23:34:36.236963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.233858    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9"} err="failed to get container status \"62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9\": rpc error: code = NotFound desc = could not find container \"62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9\": container with ID starting with 62b37f10dd3d331fb66ec3d56eed61514685d84b62d9879ccd227b52a5e519d9 not found: ID does not exist"
Sep 17 23:34:36.238031 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.238012    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-799dc7b467-d9tpj"]
Sep 17 23:34:36.594534 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.594483    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:36.621761 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.621735    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"]
Sep 17 23:34:36.622053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.622041    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="storage-initializer"
Sep 17 23:34:36.622125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.622055    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="storage-initializer"
Sep 17 23:34:36.622125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.622064    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container"
Sep 17 23:34:36.622125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.622069    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container"
Sep 17 23:34:36.622125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.622076    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kube-rbac-proxy"
Sep 17 23:34:36.622125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.622083    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kube-rbac-proxy"
Sep 17 23:34:36.622278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.622132    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kserve-container"
Sep 17 23:34:36.622278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.622140    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" containerName="kube-rbac-proxy"
Sep 17 23:34:36.625243 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.625230    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.627983 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.627968    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:34:36.628302 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.628289    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 23:34:36.628547 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.628532    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 23:34:36.628547 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.628543    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:34:36.628633 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.628580    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:34:36.628633 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.628594    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:34:36.628825 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.628812    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:34:36.635663 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.635643    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"]
Sep 17 23:34:36.728377 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.728344    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/979247c9-f60b-46d1-a957-d5187bbb97a9-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.728487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.728394    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/979247c9-f60b-46d1-a957-d5187bbb97a9-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.728487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.728428    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.728487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.728464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngr64\" (UniqueName: \"kubernetes.io/projected/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-api-access-ngr64\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.728487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.728482    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.784256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.784225    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:34:36.829440 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.829418    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngr64\" (UniqueName: \"kubernetes.io/projected/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-api-access-ngr64\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.829512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.829448    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.829512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.829485    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/979247c9-f60b-46d1-a957-d5187bbb97a9-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.829590 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.829527    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/979247c9-f60b-46d1-a957-d5187bbb97a9-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.829590 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.829553    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.829918 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.829899    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/979247c9-f60b-46d1-a957-d5187bbb97a9-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.830142 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.830126    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.830180 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.830149    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.832067 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.832048    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/979247c9-f60b-46d1-a957-d5187bbb97a9-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.839446 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.839422    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngr64\" (UniqueName: \"kubernetes.io/projected/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-api-access-ngr64\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:36.935602 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:36.935553    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:34:37.209179 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:37.209149    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"]
Sep 17 23:34:37.211702 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:34:37.211676    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979247c9_f60b_46d1_a957_d5187bbb97a9.slice/crio-47f92980ce1245a86da3739212d0f01ba53b1c43c6a9f24500d3e3c5f811b0af WatchSource:0}: Error finding container 47f92980ce1245a86da3739212d0f01ba53b1c43c6a9f24500d3e3c5f811b0af: Status 404 returned error can't find the container with id 47f92980ce1245a86da3739212d0f01ba53b1c43c6a9f24500d3e3c5f811b0af
Sep 17 23:34:38.058386 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:38.058359    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66411ed3-632a-4ec7-96f6-ae4eaaafd7d2" path="/var/lib/kubelet/pods/66411ed3-632a-4ec7-96f6-ae4eaaafd7d2/volumes"
Sep 17 23:34:38.221037 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:38.221007    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" event={"ID":"979247c9-f60b-46d1-a957-d5187bbb97a9","Type":"ContainerStarted","Data":"300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f"}
Sep 17 23:34:38.221443 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:38.221069    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" event={"ID":"979247c9-f60b-46d1-a957-d5187bbb97a9","Type":"ContainerStarted","Data":"47f92980ce1245a86da3739212d0f01ba53b1c43c6a9f24500d3e3c5f811b0af"}
Sep 17 23:34:41.235097 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:41.235068    2571 generic.go:356] "Generic (PLEG): container finished" podID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerID="300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f" exitCode=0
Sep 17 23:34:41.235412 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:34:41.235140    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" event={"ID":"979247c9-f60b-46d1-a957-d5187bbb97a9","Type":"ContainerDied","Data":"300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f"}
Sep 17 23:35:22.675047 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:35:22.674987    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:04.023434 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:04.023362    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:09.146172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:09.146140    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:09.579155 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:09.579073    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:37.900421 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:37.900350    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:36:37.914317 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:37.900644    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:36:38.799767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:38.799699    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" event={"ID":"979247c9-f60b-46d1-a957-d5187bbb97a9","Type":"ContainerStarted","Data":"d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf"}
Sep 17 23:36:38.799767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:38.799753    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" event={"ID":"979247c9-f60b-46d1-a957-d5187bbb97a9","Type":"ContainerStarted","Data":"71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e"}
Sep 17 23:36:38.799941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:38.799854    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:36:38.823172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:38.823128    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" podStartSLOduration=1.723730635 podStartE2EDuration="2m2.823114662s" totalImagesPullingTime="1m57.218097166s" totalInitContainerRuntime="3.881286861s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:38.820552368 +0000 UTC m=+1523.281470098" watchObservedRunningTime="2026-09-17 23:36:38.823114662 +0000 UTC m=+1523.284032392"
Sep 17 23:36:39.803667 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:39.803631    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:36:45.813055 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:45.813027    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:36:50.665165 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:50.665122    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:51.258039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:51.257995    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:56.441607 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:56.441572    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:36:57.103759 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:36:57.103714    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:15.829141 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:15.829102    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 23:37:25.815888 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:25.815854    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:37:30.730806 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:30.730761    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"]
Sep 17 23:37:30.731268 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:30.731076    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kserve-container" containerID="cri-o://71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e" gracePeriod=30
Sep 17 23:37:30.731268 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:30.731104    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kube-rbac-proxy" containerID="cri-o://d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf" gracePeriod=30
Sep 17 23:37:30.808066 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:30.808032    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 23:37:31.017717 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:31.017650    2571 generic.go:356] "Generic (PLEG): container finished" podID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerID="d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf" exitCode=2
Sep 17 23:37:31.017832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:31.017728    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" event={"ID":"979247c9-f60b-46d1-a957-d5187bbb97a9","Type":"ContainerDied","Data":"d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf"}
Sep 17 23:37:31.606887 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:31.606848    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:31.834959 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:31.834923    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:35.807885 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:35.807844    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 23:37:35.814292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:35.814268    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 23:37:35.927797 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:35.927760    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:35.935308 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:35.935281    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"]
Sep 17 23:37:35.937996 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:35.937974    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:35.940501 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:35.940466    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 23:37:35.940621 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:35.940533    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:37:35.948701 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:35.948680    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"]
Sep 17 23:37:36.046574 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.046548    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrft\" (UniqueName: \"kubernetes.io/projected/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-api-access-rwrft\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.046719 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.046594    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.046719 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.046636    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.046719 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.046705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.046841 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.046745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.147849 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.147822    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.147971 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.147879    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.147971 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.147912    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.147971 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.147941    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrft\" (UniqueName: \"kubernetes.io/projected/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-api-access-rwrft\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.148161 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.147990    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.148161 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:37:36.148141    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 17 23:37:36.148271 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.148197    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.148271 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:37:36.148207    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls podName:41c85844-3c31-4cc4-a1b3-a720eaf7a41a nodeName:}" failed. No retries permitted until 2026-09-17 23:37:36.648188129 +0000 UTC m=+1581.109105855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls") pod "isvc-sklearn-runtime-predictor-57599db9f-rs79h" (UID: "41c85844-3c31-4cc4-a1b3-a720eaf7a41a") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 17 23:37:36.148620 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.148598    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.148686 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.148627    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.156832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.156813    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrft\" (UniqueName: \"kubernetes.io/projected/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-api-access-rwrft\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.293776 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.293750    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:36.651989 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.651945    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.654458 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.654436    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-57599db9f-rs79h\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.848850 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.848825    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:36.933230 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:36.933201    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:37.006625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.006603    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:37:37.029145 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029120    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"]
Sep 17 23:37:37.029447 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029435    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kserve-container"
Sep 17 23:37:37.029493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029449    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kserve-container"
Sep 17 23:37:37.029493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029461    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="storage-initializer"
Sep 17 23:37:37.029493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029467    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="storage-initializer"
Sep 17 23:37:37.029493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029479    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kube-rbac-proxy"
Sep 17 23:37:37.029493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029484    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kube-rbac-proxy"
Sep 17 23:37:37.029645 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029531    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kube-rbac-proxy"
Sep 17 23:37:37.029645 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.029540    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerName="kserve-container"
Sep 17 23:37:37.031504 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.031482    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.034448 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.034426    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 23:37:37.034583 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.034430    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 23:37:37.044577 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.044552    2571 generic.go:356] "Generic (PLEG): container finished" podID="979247c9-f60b-46d1-a957-d5187bbb97a9" containerID="71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e" exitCode=0
Sep 17 23:37:37.044688 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.044636    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"
Sep 17 23:37:37.044688 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.044649    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" event={"ID":"979247c9-f60b-46d1-a957-d5187bbb97a9","Type":"ContainerDied","Data":"71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e"}
Sep 17 23:37:37.044779 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.044706    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29" event={"ID":"979247c9-f60b-46d1-a957-d5187bbb97a9","Type":"ContainerDied","Data":"47f92980ce1245a86da3739212d0f01ba53b1c43c6a9f24500d3e3c5f811b0af"}
Sep 17 23:37:37.044779 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.044745    2571 scope.go:117] "RemoveContainer" containerID="d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf"
Sep 17 23:37:37.044852 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.044823    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"]
Sep 17 23:37:37.051900 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.051864    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:37.053116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.053096    2571 scope.go:117] "RemoveContainer" containerID="71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e"
Sep 17 23:37:37.054047 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.054030    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-rbac-proxy-sar-config\") pod \"979247c9-f60b-46d1-a957-d5187bbb97a9\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") "
Sep 17 23:37:37.054156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.054069    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/979247c9-f60b-46d1-a957-d5187bbb97a9-kserve-provision-location\") pod \"979247c9-f60b-46d1-a957-d5187bbb97a9\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") "
Sep 17 23:37:37.054156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.054119    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ngr64\" (UniqueName: \"kubernetes.io/projected/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-api-access-ngr64\") pod \"979247c9-f60b-46d1-a957-d5187bbb97a9\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") "
Sep 17 23:37:37.054156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.054147    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-cabundle-cert\") pod \"979247c9-f60b-46d1-a957-d5187bbb97a9\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") "
Sep 17 23:37:37.054453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.054208    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/979247c9-f60b-46d1-a957-d5187bbb97a9-proxy-tls\") pod \"979247c9-f60b-46d1-a957-d5187bbb97a9\" (UID: \"979247c9-f60b-46d1-a957-d5187bbb97a9\") "
Sep 17 23:37:37.054520 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.054467    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "979247c9-f60b-46d1-a957-d5187bbb97a9" (UID: "979247c9-f60b-46d1-a957-d5187bbb97a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:37.054520 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.054478    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/979247c9-f60b-46d1-a957-d5187bbb97a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "979247c9-f60b-46d1-a957-d5187bbb97a9" (UID: "979247c9-f60b-46d1-a957-d5187bbb97a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:37.054825 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.054799    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "979247c9-f60b-46d1-a957-d5187bbb97a9" (UID: "979247c9-f60b-46d1-a957-d5187bbb97a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:37.056470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.056449    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/979247c9-f60b-46d1-a957-d5187bbb97a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "979247c9-f60b-46d1-a957-d5187bbb97a9" (UID: "979247c9-f60b-46d1-a957-d5187bbb97a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:37:37.056576 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.056511    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-api-access-ngr64" (OuterVolumeSpecName: "kube-api-access-ngr64") pod "979247c9-f60b-46d1-a957-d5187bbb97a9" (UID: "979247c9-f60b-46d1-a957-d5187bbb97a9"). InnerVolumeSpecName "kube-api-access-ngr64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:37.061654 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.061636    2571 scope.go:117] "RemoveContainer" containerID="300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f"
Sep 17 23:37:37.069012 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.068998    2571 scope.go:117] "RemoveContainer" containerID="d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf"
Sep 17 23:37:37.069265 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:37:37.069250    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf\": container with ID starting with d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf not found: ID does not exist" containerID="d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf"
Sep 17 23:37:37.069312 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.069271    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf"} err="failed to get container status \"d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf\": rpc error: code = NotFound desc = could not find container \"d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf\": container with ID starting with d88fafe6b33276946e69c1e3fdb9cb132c43c6ab9965f16416848fcb7ed6ebaf not found: ID does not exist"
Sep 17 23:37:37.069312 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.069286    2571 scope.go:117] "RemoveContainer" containerID="71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e"
Sep 17 23:37:37.069533 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:37:37.069517    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e\": container with ID starting with 71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e not found: ID does not exist" containerID="71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e"
Sep 17 23:37:37.069566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.069539    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e"} err="failed to get container status \"71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e\": rpc error: code = NotFound desc = could not find container \"71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e\": container with ID starting with 71bca447198cc8ff3d0fc7e0582ca8ad66739ede4c9d1d0e673e135640b4b40e not found: ID does not exist"
Sep 17 23:37:37.069566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.069553    2571 scope.go:117] "RemoveContainer" containerID="300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f"
Sep 17 23:37:37.069802 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:37:37.069786    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f\": container with ID starting with 300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f not found: ID does not exist" containerID="300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f"
Sep 17 23:37:37.069851 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.069808    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f"} err="failed to get container status \"300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f\": rpc error: code = NotFound desc = could not find container \"300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f\": container with ID starting with 300813a007e343110de53fbab78b1aa2007e51910908efb3eb89ef3bd1e0f60f not found: ID does not exist"
Sep 17 23:37:37.130886 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.130865    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"]
Sep 17 23:37:37.134283 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:37:37.134265    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c85844_3c31_4cc4_a1b3_a720eaf7a41a.slice/crio-e984baf826aef6c77fede9765859fa7cac8f6ec02c1b6e797eb926c4947f313b WatchSource:0}: Error finding container e984baf826aef6c77fede9765859fa7cac8f6ec02c1b6e797eb926c4947f313b: Status 404 returned error can't find the container with id e984baf826aef6c77fede9765859fa7cac8f6ec02c1b6e797eb926c4947f313b
Sep 17 23:37:37.155059 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260fa790-24bc-4cab-aaff-a078c643ae84-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.155158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155059    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv99s\" (UniqueName: \"kubernetes.io/projected/260fa790-24bc-4cab-aaff-a078c643ae84-kube-api-access-xv99s\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.155228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155183    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.155228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.155366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/260fa790-24bc-4cab-aaff-a078c643ae84-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.155366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155273    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/979247c9-f60b-46d1-a957-d5187bbb97a9-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:37:37.155366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155283    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:37:37.155366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155293    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/979247c9-f60b-46d1-a957-d5187bbb97a9-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:37:37.155366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155302    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngr64\" (UniqueName: \"kubernetes.io/projected/979247c9-f60b-46d1-a957-d5187bbb97a9-kube-api-access-ngr64\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:37:37.155366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.155310    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/979247c9-f60b-46d1-a957-d5187bbb97a9-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:37:37.201493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.201469    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:37.256344 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.256300    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.256463 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.256367    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.256463 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.256395    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/260fa790-24bc-4cab-aaff-a078c643ae84-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.256463 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.256444    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260fa790-24bc-4cab-aaff-a078c643ae84-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.256644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.256498    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv99s\" (UniqueName: \"kubernetes.io/projected/260fa790-24bc-4cab-aaff-a078c643ae84-kube-api-access-xv99s\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.256835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.256798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260fa790-24bc-4cab-aaff-a078c643ae84-kserve-provision-location\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.257062 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.257034    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.257174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.257064    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-cabundle-cert\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.258890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.258869    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/260fa790-24bc-4cab-aaff-a078c643ae84-proxy-tls\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.265173 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.265144    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv99s\" (UniqueName: \"kubernetes.io/projected/260fa790-24bc-4cab-aaff-a078c643ae84-kube-api-access-xv99s\") pod \"isvc-canary-create-predictor-65f7bb5954-w85j6\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.342219 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.342181    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:37.388576 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.388545    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"]
Sep 17 23:37:37.400158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.400021    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qrw29"]
Sep 17 23:37:37.598892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.598854    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:37:37.622983 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:37.622954    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"]
Sep 17 23:37:37.624303 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:37:37.624281    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260fa790_24bc_4cab_aaff_a078c643ae84.slice/crio-a4b68d388af267bf5a439d1ee4bc4ff19b7110db7d392ecb99eb62b94aa742bc WatchSource:0}: Error finding container a4b68d388af267bf5a439d1ee4bc4ff19b7110db7d392ecb99eb62b94aa742bc: Status 404 returned error can't find the container with id a4b68d388af267bf5a439d1ee4bc4ff19b7110db7d392ecb99eb62b94aa742bc
Sep 17 23:37:38.050696 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:38.050600    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" event={"ID":"260fa790-24bc-4cab-aaff-a078c643ae84","Type":"ContainerStarted","Data":"5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776"}
Sep 17 23:37:38.051125 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:38.050713    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" event={"ID":"260fa790-24bc-4cab-aaff-a078c643ae84","Type":"ContainerStarted","Data":"a4b68d388af267bf5a439d1ee4bc4ff19b7110db7d392ecb99eb62b94aa742bc"}
Sep 17 23:37:38.058680 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:38.058657    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="979247c9-f60b-46d1-a957-d5187bbb97a9" path="/var/lib/kubelet/pods/979247c9-f60b-46d1-a957-d5187bbb97a9/volumes"
Sep 17 23:37:38.058974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:38.058959    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" event={"ID":"41c85844-3c31-4cc4-a1b3-a720eaf7a41a","Type":"ContainerStarted","Data":"35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3"}
Sep 17 23:37:38.059017 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:38.059001    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" event={"ID":"41c85844-3c31-4cc4-a1b3-a720eaf7a41a","Type":"ContainerStarted","Data":"e984baf826aef6c77fede9765859fa7cac8f6ec02c1b6e797eb926c4947f313b"}
Sep 17 23:37:42.072832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:42.072740    2571 generic.go:356] "Generic (PLEG): container finished" podID="260fa790-24bc-4cab-aaff-a078c643ae84" containerID="5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776" exitCode=0
Sep 17 23:37:42.072832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:42.072778    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" event={"ID":"260fa790-24bc-4cab-aaff-a078c643ae84","Type":"ContainerDied","Data":"5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776"}
Sep 17 23:37:43.079503 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:43.079471    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" event={"ID":"260fa790-24bc-4cab-aaff-a078c643ae84","Type":"ContainerStarted","Data":"15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57"}
Sep 17 23:37:43.079865 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:43.079529    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" event={"ID":"260fa790-24bc-4cab-aaff-a078c643ae84","Type":"ContainerStarted","Data":"6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab"}
Sep 17 23:37:43.079865 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:43.079793    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:43.079973 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:43.079949    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:43.081251 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:43.081226    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:37:43.099641 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:43.099603    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podStartSLOduration=2.157048062 podStartE2EDuration="6.099591333s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.942543271s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:43.098422389 +0000 UTC m=+1587.559340122" watchObservedRunningTime="2026-09-17 23:37:43.099591333 +0000 UTC m=+1587.560509129"
Sep 17 23:37:44.085658 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:44.085626    2571 generic.go:356] "Generic (PLEG): container finished" podID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerID="35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3" exitCode=0
Sep 17 23:37:44.086078 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:44.085702    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" event={"ID":"41c85844-3c31-4cc4-a1b3-a720eaf7a41a","Type":"ContainerDied","Data":"35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3"}
Sep 17 23:37:44.086078 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:44.086026    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:37:45.092669 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:45.092633    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" event={"ID":"41c85844-3c31-4cc4-a1b3-a720eaf7a41a","Type":"ContainerStarted","Data":"8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738"}
Sep 17 23:37:45.093130 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:45.092696    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" event={"ID":"41c85844-3c31-4cc4-a1b3-a720eaf7a41a","Type":"ContainerStarted","Data":"9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81"}
Sep 17 23:37:45.093130 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:45.092906    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:45.111672 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:45.111628    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" podStartSLOduration=4.228511211 podStartE2EDuration="10.111614775s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.883103564s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:45.110517663 +0000 UTC m=+1589.571435391" watchObservedRunningTime="2026-09-17 23:37:45.111614775 +0000 UTC m=+1589.572532505"
Sep 17 23:37:46.096378 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:46.096339    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:46.097674 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:46.097643    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 23:37:47.101221 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:47.101180    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 23:37:49.090435 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:49.090408    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:37:49.090933 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:49.090903    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:37:52.105716 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:52.105686    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:37:52.106338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:52.106299    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 23:37:59.091065 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:37:59.091022    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:38:02.107632 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:02.107599    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:38:07.857193 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:07.857118    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"]
Sep 17 23:38:07.857596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:07.857373    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kserve-container" containerID="cri-o://9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81" gracePeriod=30
Sep 17 23:38:07.857596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:07.857402    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kube-rbac-proxy" containerID="cri-o://8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738" gracePeriod=30
Sep 17 23:38:07.926113 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:07.926086    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:08.209565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:08.209481    2571 generic.go:356] "Generic (PLEG): container finished" podID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerID="8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738" exitCode=2
Sep 17 23:38:08.209565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:08.209521    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" event={"ID":"41c85844-3c31-4cc4-a1b3-a720eaf7a41a","Type":"ContainerDied","Data":"8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738"}
Sep 17 23:38:08.996004 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:08.995983    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:38:09.090959 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.090925    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:38:09.115415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115389    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-rbac-proxy-sar-config\") pod \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") "
Sep 17 23:38:09.115492 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115429    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-cabundle-cert\") pod \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") "
Sep 17 23:38:09.115576 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115557    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls\") pod \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") "
Sep 17 23:38:09.115653 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115638    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kserve-provision-location\") pod \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") "
Sep 17 23:38:09.115693 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115665    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwrft\" (UniqueName: \"kubernetes.io/projected/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-api-access-rwrft\") pod \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\" (UID: \"41c85844-3c31-4cc4-a1b3-a720eaf7a41a\") "
Sep 17 23:38:09.115756 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115731    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "41c85844-3c31-4cc4-a1b3-a720eaf7a41a" (UID: "41c85844-3c31-4cc4-a1b3-a720eaf7a41a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:09.115805 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115750    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "41c85844-3c31-4cc4-a1b3-a720eaf7a41a" (UID: "41c85844-3c31-4cc4-a1b3-a720eaf7a41a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:38:09.115985 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115968    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:38:09.116036 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.115999    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:38:09.117596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.117578    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "41c85844-3c31-4cc4-a1b3-a720eaf7a41a" (UID: "41c85844-3c31-4cc4-a1b3-a720eaf7a41a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:38:09.118038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.118009    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-api-access-rwrft" (OuterVolumeSpecName: "kube-api-access-rwrft") pod "41c85844-3c31-4cc4-a1b3-a720eaf7a41a" (UID: "41c85844-3c31-4cc4-a1b3-a720eaf7a41a"). InnerVolumeSpecName "kube-api-access-rwrft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:38:09.142832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.142801    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "41c85844-3c31-4cc4-a1b3-a720eaf7a41a" (UID: "41c85844-3c31-4cc4-a1b3-a720eaf7a41a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:38:09.215675 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.215651    2571 generic.go:356] "Generic (PLEG): container finished" podID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerID="9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81" exitCode=0
Sep 17 23:38:09.215783 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.215707    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"
Sep 17 23:38:09.215783 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.215737    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" event={"ID":"41c85844-3c31-4cc4-a1b3-a720eaf7a41a","Type":"ContainerDied","Data":"9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81"}
Sep 17 23:38:09.215865 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.215795    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h" event={"ID":"41c85844-3c31-4cc4-a1b3-a720eaf7a41a","Type":"ContainerDied","Data":"e984baf826aef6c77fede9765859fa7cac8f6ec02c1b6e797eb926c4947f313b"}
Sep 17 23:38:09.215865 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.215823    2571 scope.go:117] "RemoveContainer" containerID="8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738"
Sep 17 23:38:09.216364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.216319    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:38:09.216364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.216354    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:38:09.216490 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.216371    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrft\" (UniqueName: \"kubernetes.io/projected/41c85844-3c31-4cc4-a1b3-a720eaf7a41a-kube-api-access-rwrft\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:38:09.224806 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.224791    2571 scope.go:117] "RemoveContainer" containerID="9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81"
Sep 17 23:38:09.232064 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.232043    2571 scope.go:117] "RemoveContainer" containerID="35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3"
Sep 17 23:38:09.239314 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.239293    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"]
Sep 17 23:38:09.239878 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.239865    2571 scope.go:117] "RemoveContainer" containerID="8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738"
Sep 17 23:38:09.240088 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:38:09.240070    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738\": container with ID starting with 8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738 not found: ID does not exist" containerID="8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738"
Sep 17 23:38:09.240137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.240096    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738"} err="failed to get container status \"8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738\": rpc error: code = NotFound desc = could not find container \"8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738\": container with ID starting with 8fd54804e76cc7062eb75a89a1ff4d3fc79d00a2cf9780a66726ea31fc779738 not found: ID does not exist"
Sep 17 23:38:09.240137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.240115    2571 scope.go:117] "RemoveContainer" containerID="9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81"
Sep 17 23:38:09.240345 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:38:09.240312    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81\": container with ID starting with 9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81 not found: ID does not exist" containerID="9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81"
Sep 17 23:38:09.240391 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.240355    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81"} err="failed to get container status \"9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81\": rpc error: code = NotFound desc = could not find container \"9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81\": container with ID starting with 9b9b3241ff84b86b6ec9f9fee2e194946367fc548758ccf74cac9c65f5e15e81 not found: ID does not exist"
Sep 17 23:38:09.240391 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.240375    2571 scope.go:117] "RemoveContainer" containerID="35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3"
Sep 17 23:38:09.240583 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:38:09.240569    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3\": container with ID starting with 35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3 not found: ID does not exist" containerID="35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3"
Sep 17 23:38:09.240622 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.240587    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3"} err="failed to get container status \"35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3\": rpc error: code = NotFound desc = could not find container \"35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3\": container with ID starting with 35cd6c610485fea062f6fdafa0a93fe285d4e4de20464a12babfaf36c2f306d3 not found: ID does not exist"
Sep 17 23:38:09.244297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:09.244279    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-57599db9f-rs79h"]
Sep 17 23:38:10.057824 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:10.057794    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" path="/var/lib/kubelet/pods/41c85844-3c31-4cc4-a1b3-a720eaf7a41a/volumes"
Sep 17 23:38:13.014607 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.014569    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:13.032496 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.032470    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"]
Sep 17 23:38:13.032946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.032926    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kserve-container"
Sep 17 23:38:13.032995 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.032952    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kserve-container"
Sep 17 23:38:13.032995 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.032966    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kube-rbac-proxy"
Sep 17 23:38:13.032995 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.032975    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kube-rbac-proxy"
Sep 17 23:38:13.033099 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.032996    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="storage-initializer"
Sep 17 23:38:13.033099 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.033005    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="storage-initializer"
Sep 17 23:38:13.033099 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.033075    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kserve-container"
Sep 17 23:38:13.033099 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.033088    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="41c85844-3c31-4cc4-a1b3-a720eaf7a41a" containerName="kube-rbac-proxy"
Sep 17 23:38:13.036627 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.036610    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.039004 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.038986    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:38:13.039004 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.038997    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 23:38:13.041369 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.041350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.041457 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.041397    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.041457 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.041423    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.041457 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.041446    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z497b\" (UniqueName: \"kubernetes.io/projected/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-api-access-z497b\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.041563 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.041517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.044866 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.044843    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"]
Sep 17 23:38:13.141742 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.141720    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.141832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.141758    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.141832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.141786    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.141832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.141801    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.141832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.141816    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z497b\" (UniqueName: \"kubernetes.io/projected/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-api-access-z497b\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.142021 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:38:13.141911    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 23:38:13.142021 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:38:13.141967    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls podName:a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7 nodeName:}" failed. No retries permitted until 2026-09-17 23:38:13.641949494 +0000 UTC m=+1618.102867218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" (UID: "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Sep 17 23:38:13.142176 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.142147    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.142469 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.142450    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.142511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.142465    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.150367 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.150350    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z497b\" (UniqueName: \"kubernetes.io/projected/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-api-access-z497b\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.387021 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.386986    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:38:13.645584 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.645525    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.647726 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.647708    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:13.947859 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:13.947810    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:14.217566 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:14.217541    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"]
Sep 17 23:38:14.220135 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:38:14.220111    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cf64e3_2142_4d53_b9a2_ac55ec1b1ca7.slice/crio-7fe95f6737b259e2a4c7397aa93be87fe27317f089e5530583d127d7df51d3e3 WatchSource:0}: Error finding container 7fe95f6737b259e2a4c7397aa93be87fe27317f089e5530583d127d7df51d3e3: Status 404 returned error can't find the container with id 7fe95f6737b259e2a4c7397aa93be87fe27317f089e5530583d127d7df51d3e3
Sep 17 23:38:14.240901 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:14.240879    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" event={"ID":"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7","Type":"ContainerStarted","Data":"7fe95f6737b259e2a4c7397aa93be87fe27317f089e5530583d127d7df51d3e3"}
Sep 17 23:38:15.246736 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:15.246704    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" event={"ID":"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7","Type":"ContainerStarted","Data":"19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103"}
Sep 17 23:38:18.259569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:18.259504    2571 generic.go:356] "Generic (PLEG): container finished" podID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerID="19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103" exitCode=0
Sep 17 23:38:18.259896 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:18.259576    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" event={"ID":"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7","Type":"ContainerDied","Data":"19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103"}
Sep 17 23:38:19.091501 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:19.091464    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:38:19.266105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:19.266069    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" event={"ID":"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7","Type":"ContainerStarted","Data":"c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496"}
Sep 17 23:38:19.266575 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:19.266178    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" event={"ID":"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7","Type":"ContainerStarted","Data":"3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff"}
Sep 17 23:38:19.287582 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:19.287538    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" podStartSLOduration=2.591989819 podStartE2EDuration="6.287524819s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.695535s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:38:19.285715324 +0000 UTC m=+1623.746633066" watchObservedRunningTime="2026-09-17 23:38:19.287524819 +0000 UTC m=+1623.748442549"
Sep 17 23:38:24.267120 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:24.267088    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:24.267692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:24.267549    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:24.271808 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:24.271784    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:29.091548 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:29.091514    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:38:39.091697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:39.091657    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:38:49.091442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:49.091404    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:38:55.293309 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:55.293274    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:38:59.091414 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:38:59.091389    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:39:01.831422 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:01.831387    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:03.114551 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:03.114515    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"]
Sep 17 23:39:03.114994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:03.114804    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kserve-container" containerID="cri-o://3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff" gracePeriod=30
Sep 17 23:39:03.114994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:03.114844    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kube-rbac-proxy" containerID="cri-o://c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496" gracePeriod=30
Sep 17 23:39:03.135184 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:03.135150    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:03.448701 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:03.448633    2571 generic.go:356] "Generic (PLEG): container finished" podID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerID="c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496" exitCode=2
Sep 17 23:39:03.448804 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:03.448703    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" event={"ID":"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7","Type":"ContainerDied","Data":"c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496"}
Sep 17 23:39:04.272238 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:04.272203    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 17 23:39:08.281373 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.281317    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:08.346418 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.346394    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:39:08.432361 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.432279    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-rbac-proxy-sar-config\") pod \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") "
Sep 17 23:39:08.432474 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.432378    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls\") pod \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") "
Sep 17 23:39:08.432474 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.432425    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-cabundle-cert\") pod \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") "
Sep 17 23:39:08.432603 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.432556    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z497b\" (UniqueName: \"kubernetes.io/projected/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-api-access-z497b\") pod \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") "
Sep 17 23:39:08.432657 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.432612    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kserve-provision-location\") pod \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\" (UID: \"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7\") "
Sep 17 23:39:08.432712 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.432696    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" (UID: "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:08.432787 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.432767    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" (UID: "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:08.432930 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.432911    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" (UID: "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:08.433235 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.433156    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:08.433277 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.433243    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:08.433277 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.433257    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:08.434623 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.434592    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" (UID: "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:39:08.434720 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.434646    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-api-access-z497b" (OuterVolumeSpecName: "kube-api-access-z497b") pod "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" (UID: "a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7"). InnerVolumeSpecName "kube-api-access-z497b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:08.469852 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.469826    2571 generic.go:356] "Generic (PLEG): container finished" podID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerID="3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff" exitCode=0
Sep 17 23:39:08.469944 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.469880    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"
Sep 17 23:39:08.469944 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.469902    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" event={"ID":"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7","Type":"ContainerDied","Data":"3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff"}
Sep 17 23:39:08.470020 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.469953    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh" event={"ID":"a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7","Type":"ContainerDied","Data":"7fe95f6737b259e2a4c7397aa93be87fe27317f089e5530583d127d7df51d3e3"}
Sep 17 23:39:08.470020 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.469981    2571 scope.go:117] "RemoveContainer" containerID="c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496"
Sep 17 23:39:08.477890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.477876    2571 scope.go:117] "RemoveContainer" containerID="3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff"
Sep 17 23:39:08.484904 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.484890    2571 scope.go:117] "RemoveContainer" containerID="19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103"
Sep 17 23:39:08.491989 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.491968    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:08.492720 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.492699    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"]
Sep 17 23:39:08.493003 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.492985    2571 scope.go:117] "RemoveContainer" containerID="c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496"
Sep 17 23:39:08.493271 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:08.493249    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496\": container with ID starting with c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496 not found: ID does not exist" containerID="c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496"
Sep 17 23:39:08.493389 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.493278    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496"} err="failed to get container status \"c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496\": rpc error: code = NotFound desc = could not find container \"c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496\": container with ID starting with c4f33e3cf899ede47ffdf3bde8cc878925e41d8204bfd82190d3970518b11496 not found: ID does not exist"
Sep 17 23:39:08.493389 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.493293    2571 scope.go:117] "RemoveContainer" containerID="3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff"
Sep 17 23:39:08.493702 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:08.493677    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff\": container with ID starting with 3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff not found: ID does not exist" containerID="3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff"
Sep 17 23:39:08.493821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.493708    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff"} err="failed to get container status \"3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff\": rpc error: code = NotFound desc = could not find container \"3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff\": container with ID starting with 3d0c320787f00778dca7b676b63b4fb6f5e1b3fdd962233772624126950bdaff not found: ID does not exist"
Sep 17 23:39:08.493821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.493727    2571 scope.go:117] "RemoveContainer" containerID="19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103"
Sep 17 23:39:08.494576 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:08.494549    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103\": container with ID starting with 19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103 not found: ID does not exist" containerID="19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103"
Sep 17 23:39:08.494664 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.494585    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103"} err="failed to get container status \"19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103\": rpc error: code = NotFound desc = could not find container \"19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103\": container with ID starting with 19c2aaa0fc7de621b70f39199ec1722e4e1cb1f081e0a12bb2783dfc19b35103 not found: ID does not exist"
Sep 17 23:39:08.495721 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.495705    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-7bfhh"]
Sep 17 23:39:08.534205 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.534189    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:08.534286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:08.534207    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z497b\" (UniqueName: \"kubernetes.io/projected/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7-kube-api-access-z497b\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:10.057188 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:10.057158    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" path="/var/lib/kubelet/pods/a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7/volumes"
Sep 17 23:39:17.004487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:17.004457    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"]
Sep 17 23:39:17.004857 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:17.004680    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" containerID="cri-o://6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab" gracePeriod=30
Sep 17 23:39:17.004857 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:17.004783    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kube-rbac-proxy" containerID="cri-o://15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57" gracePeriod=30
Sep 17 23:39:17.048207 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:17.048186    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:17.332478 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:17.332400    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:17.511428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:17.511392    2571 generic.go:356] "Generic (PLEG): container finished" podID="260fa790-24bc-4cab-aaff-a078c643ae84" containerID="15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57" exitCode=2
Sep 17 23:39:17.511560 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:17.511468    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" event={"ID":"260fa790-24bc-4cab-aaff-a078c643ae84","Type":"ContainerDied","Data":"15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57"}
Sep 17 23:39:19.087209 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:19.087170    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 23:39:19.091531 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:19.091508    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 23:39:20.743800 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.743777    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:39:20.821433 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.821378    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260fa790-24bc-4cab-aaff-a078c643ae84-kserve-provision-location\") pod \"260fa790-24bc-4cab-aaff-a078c643ae84\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") "
Sep 17 23:39:20.821433 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.821409    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-kube-rbac-proxy-sar-config\") pod \"260fa790-24bc-4cab-aaff-a078c643ae84\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") "
Sep 17 23:39:20.821586 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.821492    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/260fa790-24bc-4cab-aaff-a078c643ae84-proxy-tls\") pod \"260fa790-24bc-4cab-aaff-a078c643ae84\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") "
Sep 17 23:39:20.821586 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.821510    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-cabundle-cert\") pod \"260fa790-24bc-4cab-aaff-a078c643ae84\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") "
Sep 17 23:39:20.821586 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.821536    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv99s\" (UniqueName: \"kubernetes.io/projected/260fa790-24bc-4cab-aaff-a078c643ae84-kube-api-access-xv99s\") pod \"260fa790-24bc-4cab-aaff-a078c643ae84\" (UID: \"260fa790-24bc-4cab-aaff-a078c643ae84\") "
Sep 17 23:39:20.821740 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.821691    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260fa790-24bc-4cab-aaff-a078c643ae84-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "260fa790-24bc-4cab-aaff-a078c643ae84" (UID: "260fa790-24bc-4cab-aaff-a078c643ae84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:20.821832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.821814    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "260fa790-24bc-4cab-aaff-a078c643ae84" (UID: "260fa790-24bc-4cab-aaff-a078c643ae84"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:20.821916 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.821895    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "260fa790-24bc-4cab-aaff-a078c643ae84" (UID: "260fa790-24bc-4cab-aaff-a078c643ae84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:20.823664 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.823634    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260fa790-24bc-4cab-aaff-a078c643ae84-kube-api-access-xv99s" (OuterVolumeSpecName: "kube-api-access-xv99s") pod "260fa790-24bc-4cab-aaff-a078c643ae84" (UID: "260fa790-24bc-4cab-aaff-a078c643ae84"). InnerVolumeSpecName "kube-api-access-xv99s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:20.823744 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.823724    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260fa790-24bc-4cab-aaff-a078c643ae84-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "260fa790-24bc-4cab-aaff-a078c643ae84" (UID: "260fa790-24bc-4cab-aaff-a078c643ae84"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:39:20.921996 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.921968    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/260fa790-24bc-4cab-aaff-a078c643ae84-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:20.921996 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.921993    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:20.922133 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.922006    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv99s\" (UniqueName: \"kubernetes.io/projected/260fa790-24bc-4cab-aaff-a078c643ae84-kube-api-access-xv99s\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:20.922133 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.922020    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260fa790-24bc-4cab-aaff-a078c643ae84-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:20.922133 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:20.922032    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/260fa790-24bc-4cab-aaff-a078c643ae84-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:39:21.527846 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.527816    2571 generic.go:356] "Generic (PLEG): container finished" podID="260fa790-24bc-4cab-aaff-a078c643ae84" containerID="6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab" exitCode=0
Sep 17 23:39:21.527963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.527870    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"
Sep 17 23:39:21.527963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.527901    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" event={"ID":"260fa790-24bc-4cab-aaff-a078c643ae84","Type":"ContainerDied","Data":"6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab"}
Sep 17 23:39:21.527963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.527957    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6" event={"ID":"260fa790-24bc-4cab-aaff-a078c643ae84","Type":"ContainerDied","Data":"a4b68d388af267bf5a439d1ee4bc4ff19b7110db7d392ecb99eb62b94aa742bc"}
Sep 17 23:39:21.528101 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.527982    2571 scope.go:117] "RemoveContainer" containerID="15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57"
Sep 17 23:39:21.536146 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.536130    2571 scope.go:117] "RemoveContainer" containerID="6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab"
Sep 17 23:39:21.543415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.543399    2571 scope.go:117] "RemoveContainer" containerID="5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776"
Sep 17 23:39:21.550510 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.550491    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"]
Sep 17 23:39:21.550886 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.550871    2571 scope.go:117] "RemoveContainer" containerID="15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57"
Sep 17 23:39:21.551130 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:21.551111    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57\": container with ID starting with 15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57 not found: ID does not exist" containerID="15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57"
Sep 17 23:39:21.551203 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.551141    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57"} err="failed to get container status \"15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57\": rpc error: code = NotFound desc = could not find container \"15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57\": container with ID starting with 15cdd9eec4f307fe078257687ab669701a140762685a5d660a1895b3c339ed57 not found: ID does not exist"
Sep 17 23:39:21.551203 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.551180    2571 scope.go:117] "RemoveContainer" containerID="6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab"
Sep 17 23:39:21.551470 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:21.551454    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab\": container with ID starting with 6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab not found: ID does not exist" containerID="6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab"
Sep 17 23:39:21.551519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.551475    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab"} err="failed to get container status \"6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab\": rpc error: code = NotFound desc = could not find container \"6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab\": container with ID starting with 6b27e9325e3a847af417f077dcd8087b56a978a21a5409a9d37d1601e1f6dcab not found: ID does not exist"
Sep 17 23:39:21.551519 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.551490    2571 scope.go:117] "RemoveContainer" containerID="5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776"
Sep 17 23:39:21.551730 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:21.551713    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776\": container with ID starting with 5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776 not found: ID does not exist" containerID="5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776"
Sep 17 23:39:21.551777 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.551738    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776"} err="failed to get container status \"5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776\": rpc error: code = NotFound desc = could not find container \"5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776\": container with ID starting with 5a5a6d5f986e8c9e0221f5f26084ece93b469be76764f7aba2de3d86ca1f4776 not found: ID does not exist"
Sep 17 23:39:21.554835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:21.554814    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-65f7bb5954-w85j6"]
Sep 17 23:39:22.058115 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.058089    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" path="/var/lib/kubelet/pods/260fa790-24bc-4cab-aaff-a078c643ae84/volumes"
Sep 17 23:39:22.173178 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.173156    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:22.176038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176007    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"]
Sep 17 23:39:22.176294 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176283    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="storage-initializer"
Sep 17 23:39:22.176372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176297    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="storage-initializer"
Sep 17 23:39:22.176372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176306    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kube-rbac-proxy"
Sep 17 23:39:22.176372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176311    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kube-rbac-proxy"
Sep 17 23:39:22.176372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176341    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kserve-container"
Sep 17 23:39:22.176372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176350    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kserve-container"
Sep 17 23:39:22.176372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176358    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kube-rbac-proxy"
Sep 17 23:39:22.176372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176363    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kube-rbac-proxy"
Sep 17 23:39:22.176372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176373    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="storage-initializer"
Sep 17 23:39:22.176634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176379    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="storage-initializer"
Sep 17 23:39:22.176634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176385    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container"
Sep 17 23:39:22.176634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176390    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container"
Sep 17 23:39:22.176634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176430    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kserve-container"
Sep 17 23:39:22.176634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176436    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kserve-container"
Sep 17 23:39:22.176634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176443    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7cf64e3-2142-4d53-b9a2-ac55ec1b1ca7" containerName="kube-rbac-proxy"
Sep 17 23:39:22.176634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.176451    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="260fa790-24bc-4cab-aaff-a078c643ae84" containerName="kube-rbac-proxy"
Sep 17 23:39:22.180756 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.180743    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.186932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.186626    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:39:22.186932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.186689    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:39:22.186932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.186931    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:39:22.187150 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.186950    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:39:22.187386 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.187308    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 23:39:22.187702 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.187679    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 23:39:22.187702 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.187693    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:39:22.192307 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.192288    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"]
Sep 17 23:39:22.266550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.266530    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"]
Sep 17 23:39:22.269772 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.269754    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.272137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.272119    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 23:39:22.278171 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.278141    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:22.278901 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.278882    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"]
Sep 17 23:39:22.331943 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.331891    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.331943 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.331930    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.332046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.331948    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.332046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.331968    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.332046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.332004    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpjb\" (UniqueName: \"kubernetes.io/projected/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-api-access-5mpjb\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.432299 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432278    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.432413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432310    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.432413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.432413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432367    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed3a6c83-503b-422b-b145-cbed71089a83-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.432413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432383    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.432413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432400    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.432634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpjb\" (UniqueName: \"kubernetes.io/projected/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-api-access-5mpjb\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.432634 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:22.432425    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 23:39:22.432634 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:22.432485    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls podName:a5f6254d-d80a-4524-86e3-c3dac1960aa3 nodeName:}" failed. No retries permitted until 2026-09-17 23:39:22.932467033 +0000 UTC m=+1687.393384757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls") pod "isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" (UID: "a5f6254d-d80a-4524-86e3-c3dac1960aa3") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 23:39:22.432634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432524    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4jmc\" (UniqueName: \"kubernetes.io/projected/ed3a6c83-503b-422b-b145-cbed71089a83-kube-api-access-z4jmc\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.432634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432559    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.432634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432597    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.432844 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.432997 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.432981    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.433058 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.433040    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.441469 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.441443    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpjb\" (UniqueName: \"kubernetes.io/projected/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-api-access-5mpjb\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.508666 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.508643    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:22.532905 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.532880    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4jmc\" (UniqueName: \"kubernetes.io/projected/ed3a6c83-503b-422b-b145-cbed71089a83-kube-api-access-z4jmc\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.532987 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.532929    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.532987 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.532960    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.533085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.533006    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed3a6c83-503b-422b-b145-cbed71089a83-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.533085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.533030    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.533185 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:22.533103    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 23:39:22.533185 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:39:22.533162    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls podName:ed3a6c83-503b-422b-b145-cbed71089a83 nodeName:}" failed. No retries permitted until 2026-09-17 23:39:23.033146209 +0000 UTC m=+1687.494063917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls") pod "isvc-canary-promote-predictor-fc78f5cb6-sdrjn" (UID: "ed3a6c83-503b-422b-b145-cbed71089a83") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 17 23:39:22.533453 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.533435    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed3a6c83-503b-422b-b145-cbed71089a83-kserve-provision-location\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.533631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.533615    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.533669 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.533633    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-cabundle-cert\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.541681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.541661    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4jmc\" (UniqueName: \"kubernetes.io/projected/ed3a6c83-503b-422b-b145-cbed71089a83-kube-api-access-z4jmc\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:22.632450 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.632426    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:22.934852 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.934809    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:22.936955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:22.936934    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8555cb9565-kv5dv\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:23.035396 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.035371    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:23.037409 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.037394    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls\") pod \"isvc-canary-promote-predictor-fc78f5cb6-sdrjn\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:23.091029 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.091007    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:23.179415 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.179388    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:23.366636 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.366605    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"]
Sep 17 23:39:23.367633 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:39:23.367609    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f6254d_d80a_4524_86e3_c3dac1960aa3.slice/crio-034c81f6cc72969126b24cb951b8d0b32f943d22d60948fc3dc78fa99ef28988 WatchSource:0}: Error finding container 034c81f6cc72969126b24cb951b8d0b32f943d22d60948fc3dc78fa99ef28988: Status 404 returned error can't find the container with id 034c81f6cc72969126b24cb951b8d0b32f943d22d60948fc3dc78fa99ef28988
Sep 17 23:39:23.449160 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.449101    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"]
Sep 17 23:39:23.453608 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:39:23.453584    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3a6c83_503b_422b_b145_cbed71089a83.slice/crio-f3e733e70c5775aa20c189de3563a897b57e1ce93b98b45fc68bbe9e32a6a7fe WatchSource:0}: Error finding container f3e733e70c5775aa20c189de3563a897b57e1ce93b98b45fc68bbe9e32a6a7fe: Status 404 returned error can't find the container with id f3e733e70c5775aa20c189de3563a897b57e1ce93b98b45fc68bbe9e32a6a7fe
Sep 17 23:39:23.538724 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.538688    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" event={"ID":"ed3a6c83-503b-422b-b145-cbed71089a83","Type":"ContainerStarted","Data":"e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33"}
Sep 17 23:39:23.538847 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.538762    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" event={"ID":"ed3a6c83-503b-422b-b145-cbed71089a83","Type":"ContainerStarted","Data":"f3e733e70c5775aa20c189de3563a897b57e1ce93b98b45fc68bbe9e32a6a7fe"}
Sep 17 23:39:23.540493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.540467    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" event={"ID":"a5f6254d-d80a-4524-86e3-c3dac1960aa3","Type":"ContainerStarted","Data":"5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d"}
Sep 17 23:39:23.540587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:23.540529    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" event={"ID":"a5f6254d-d80a-4524-86e3-c3dac1960aa3","Type":"ContainerStarted","Data":"034c81f6cc72969126b24cb951b8d0b32f943d22d60948fc3dc78fa99ef28988"}
Sep 17 23:39:27.557590 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:27.557511    2571 generic.go:356] "Generic (PLEG): container finished" podID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerID="5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d" exitCode=0
Sep 17 23:39:27.557590 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:27.557567    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" event={"ID":"a5f6254d-d80a-4524-86e3-c3dac1960aa3","Type":"ContainerDied","Data":"5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d"}
Sep 17 23:39:27.992787 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:27.992757    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:39:28.562592 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:28.562562    2571 generic.go:356] "Generic (PLEG): container finished" podID="ed3a6c83-503b-422b-b145-cbed71089a83" containerID="e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33" exitCode=0
Sep 17 23:39:28.562966 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:28.562637    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" event={"ID":"ed3a6c83-503b-422b-b145-cbed71089a83","Type":"ContainerDied","Data":"e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33"}
Sep 17 23:39:28.564843 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:28.564824    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" event={"ID":"a5f6254d-d80a-4524-86e3-c3dac1960aa3","Type":"ContainerStarted","Data":"ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03"}
Sep 17 23:39:28.564958 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:28.564864    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" event={"ID":"a5f6254d-d80a-4524-86e3-c3dac1960aa3","Type":"ContainerStarted","Data":"a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4"}
Sep 17 23:39:28.565058 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:28.565036    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:28.565158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:28.565090    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:28.566360 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:28.566315    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:39:28.601167 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:28.601131    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podStartSLOduration=2.812689585 podStartE2EDuration="6.601114491s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788424906s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:39:28.599533643 +0000 UTC m=+1693.060451373" watchObservedRunningTime="2026-09-17 23:39:28.601114491 +0000 UTC m=+1693.062032222"
Sep 17 23:39:29.570594 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:29.570562    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" event={"ID":"ed3a6c83-503b-422b-b145-cbed71089a83","Type":"ContainerStarted","Data":"09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f"}
Sep 17 23:39:29.570972 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:29.570618    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" event={"ID":"ed3a6c83-503b-422b-b145-cbed71089a83","Type":"ContainerStarted","Data":"4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17"}
Sep 17 23:39:29.571044 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:29.570982    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:39:29.571097 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:29.571055    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:29.572064 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:29.572041    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:39:29.590722 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:29.590677    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podStartSLOduration=3.536386415 podStartE2EDuration="7.590662237s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.054275822s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:39:29.590020711 +0000 UTC m=+1694.050938441" watchObservedRunningTime="2026-09-17 23:39:29.590662237 +0000 UTC m=+1694.051579979"
Sep 17 23:39:30.575051 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:30.575014    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:30.575424 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:30.575170    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:39:31.579310 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:31.579273    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:39:34.575473 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:34.575448    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:39:34.575986 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:34.575964    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:39:36.583212 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:36.583187    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:39:36.583799 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:36.583774    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:39:44.576830 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:44.576757    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:39:46.583947 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:46.583911    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:39:54.576078 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:54.576039    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:39:56.584019 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:39:56.583984    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:40:04.576628 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:04.576591    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:40:06.584123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:06.584082    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:40:14.576162 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:14.576121    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:40:16.584180 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:16.584153    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:40:24.575910 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:24.575867    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:40:26.583725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:26.583685    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 23:40:34.576837 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:34.576800    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:40:36.585070 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:36.585045    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:40:38.509372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:38.509312    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:43.625848 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:43.625810    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:43.725715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:43.725682    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:44.577298 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:44.577275    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:40:52.262090 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:52.262059    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"]
Sep 17 23:40:52.262491 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:52.262269    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container" containerID="cri-o://4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17" gracePeriod=30
Sep 17 23:40:52.262491 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:52.262342    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kube-rbac-proxy" containerID="cri-o://09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f" gracePeriod=30
Sep 17 23:40:52.287135 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:52.287113    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:40:52.898278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:52.898249    2571 generic.go:356] "Generic (PLEG): container finished" podID="ed3a6c83-503b-422b-b145-cbed71089a83" containerID="09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f" exitCode=2
Sep 17 23:40:52.898445 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:52.898337    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" event={"ID":"ed3a6c83-503b-422b-b145-cbed71089a83","Type":"ContainerDied","Data":"09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f"}
Sep 17 23:40:55.799302 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.799282    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:40:55.873556 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.873533    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls\") pod \"ed3a6c83-503b-422b-b145-cbed71089a83\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") "
Sep 17 23:40:55.873681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.873565    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-cabundle-cert\") pod \"ed3a6c83-503b-422b-b145-cbed71089a83\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") "
Sep 17 23:40:55.873681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.873614    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-kube-rbac-proxy-sar-config\") pod \"ed3a6c83-503b-422b-b145-cbed71089a83\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") "
Sep 17 23:40:55.873681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.873647    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4jmc\" (UniqueName: \"kubernetes.io/projected/ed3a6c83-503b-422b-b145-cbed71089a83-kube-api-access-z4jmc\") pod \"ed3a6c83-503b-422b-b145-cbed71089a83\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") "
Sep 17 23:40:55.873681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.873674    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed3a6c83-503b-422b-b145-cbed71089a83-kserve-provision-location\") pod \"ed3a6c83-503b-422b-b145-cbed71089a83\" (UID: \"ed3a6c83-503b-422b-b145-cbed71089a83\") "
Sep 17 23:40:55.874023 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.873996    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed3a6c83-503b-422b-b145-cbed71089a83" (UID: "ed3a6c83-503b-422b-b145-cbed71089a83"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:40:55.874023 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.874010    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3a6c83-503b-422b-b145-cbed71089a83-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed3a6c83-503b-422b-b145-cbed71089a83" (UID: "ed3a6c83-503b-422b-b145-cbed71089a83"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:40:55.874250 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.873999    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed3a6c83-503b-422b-b145-cbed71089a83" (UID: "ed3a6c83-503b-422b-b145-cbed71089a83"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:40:55.875689 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.875666    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed3a6c83-503b-422b-b145-cbed71089a83" (UID: "ed3a6c83-503b-422b-b145-cbed71089a83"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:40:55.875786 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.875767    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3a6c83-503b-422b-b145-cbed71089a83-kube-api-access-z4jmc" (OuterVolumeSpecName: "kube-api-access-z4jmc") pod "ed3a6c83-503b-422b-b145-cbed71089a83" (UID: "ed3a6c83-503b-422b-b145-cbed71089a83"). InnerVolumeSpecName "kube-api-access-z4jmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:40:55.911820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.911795    2571 generic.go:356] "Generic (PLEG): container finished" podID="ed3a6c83-503b-422b-b145-cbed71089a83" containerID="4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17" exitCode=0
Sep 17 23:40:55.911913 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.911838    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" event={"ID":"ed3a6c83-503b-422b-b145-cbed71089a83","Type":"ContainerDied","Data":"4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17"}
Sep 17 23:40:55.911913 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.911856    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"
Sep 17 23:40:55.911913 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.911897    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn" event={"ID":"ed3a6c83-503b-422b-b145-cbed71089a83","Type":"ContainerDied","Data":"f3e733e70c5775aa20c189de3563a897b57e1ce93b98b45fc68bbe9e32a6a7fe"}
Sep 17 23:40:55.912073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.911935    2571 scope.go:117] "RemoveContainer" containerID="09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f"
Sep 17 23:40:55.920050 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.920026    2571 scope.go:117] "RemoveContainer" containerID="4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17"
Sep 17 23:40:55.927342 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.927310    2571 scope.go:117] "RemoveContainer" containerID="e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33"
Sep 17 23:40:55.934377 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.934357    2571 scope.go:117] "RemoveContainer" containerID="09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f"
Sep 17 23:40:55.934632 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:40:55.934615    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f\": container with ID starting with 09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f not found: ID does not exist" containerID="09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f"
Sep 17 23:40:55.934687 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.934639    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f"} err="failed to get container status \"09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f\": rpc error: code = NotFound desc = could not find container \"09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f\": container with ID starting with 09ce7ad344dc205727faabe13b19b694675bc5ebc4b11eee605348ad39a1755f not found: ID does not exist"
Sep 17 23:40:55.934687 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.934655    2571 scope.go:117] "RemoveContainer" containerID="4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17"
Sep 17 23:40:55.934878 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:40:55.934863    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17\": container with ID starting with 4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17 not found: ID does not exist" containerID="4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17"
Sep 17 23:40:55.934921 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.934882    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17"} err="failed to get container status \"4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17\": rpc error: code = NotFound desc = could not find container \"4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17\": container with ID starting with 4d572b26767e8c876b953e37ff79bebf85d7ba20e9630c63aa6b7ee6c6788e17 not found: ID does not exist"
Sep 17 23:40:55.934921 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.934894    2571 scope.go:117] "RemoveContainer" containerID="e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33"
Sep 17 23:40:55.935014 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.934983    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"]
Sep 17 23:40:55.935164 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:40:55.935146    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33\": container with ID starting with e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33 not found: ID does not exist" containerID="e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33"
Sep 17 23:40:55.935210 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.935169    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33"} err="failed to get container status \"e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33\": rpc error: code = NotFound desc = could not find container \"e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33\": container with ID starting with e8d62005aaa2e8350adbbbaa638d0f3347a584c8da872a69a5eda0c2919a7c33 not found: ID does not exist"
Sep 17 23:40:55.939090 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.939071    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-fc78f5cb6-sdrjn"]
Sep 17 23:40:55.974647 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.974627    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed3a6c83-503b-422b-b145-cbed71089a83-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:40:55.974720 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.974648    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:40:55.974720 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.974658    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed3a6c83-503b-422b-b145-cbed71089a83-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:40:55.974720 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.974666    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4jmc\" (UniqueName: \"kubernetes.io/projected/ed3a6c83-503b-422b-b145-cbed71089a83-kube-api-access-z4jmc\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:40:55.974720 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:55.974675    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed3a6c83-503b-422b-b145-cbed71089a83-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:40:56.058060 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:40:56.058036    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" path="/var/lib/kubelet/pods/ed3a6c83-503b-422b-b145-cbed71089a83/volumes"
Sep 17 23:41:37.927531 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:41:37.927428    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:41:37.928740 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:41:37.928716    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:42:13.813944 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:13.813907    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:18.971410 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.971371    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:18.972353 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972310    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"]
Sep 17 23:42:18.972618 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972604    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="storage-initializer"
Sep 17 23:42:18.972660 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972620    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="storage-initializer"
Sep 17 23:42:18.972660 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972634    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container"
Sep 17 23:42:18.972660 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972639    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container"
Sep 17 23:42:18.972660 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972644    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kube-rbac-proxy"
Sep 17 23:42:18.972660 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972649    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kube-rbac-proxy"
Sep 17 23:42:18.972808 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972743    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kube-rbac-proxy"
Sep 17 23:42:18.972808 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.972757    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3a6c83-503b-422b-b145-cbed71089a83" containerName="kserve-container"
Sep 17 23:42:18.975694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.975677    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:18.978123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.978102    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 17 23:42:18.978237 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.978102    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 17 23:42:18.987095 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:18.987077    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"]
Sep 17 23:42:19.108941 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.108918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:19.125852 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.125832    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24frx\" (UniqueName: \"kubernetes.io/projected/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-api-access-24frx\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.125919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.125863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.125919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.125890    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.125919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.125912    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.126021 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.125958    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.226700 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.226654    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24frx\" (UniqueName: \"kubernetes.io/projected/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-api-access-24frx\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.226700 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.226694    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.226822 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.226718    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.226822 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.226748    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.226822 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.226774    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.227140 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.227121    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.227400 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.227381    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.227477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.227459    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.229246 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.229230    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.235994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.235971    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24frx\" (UniqueName: \"kubernetes.io/projected/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-api-access-24frx\") pod \"isvc-tensorflow-predictor-547dfc5669-qw5s8\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.286118 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.286095    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:19.560768 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:19.560687    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"]
Sep 17 23:42:19.563083 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:42:19.563051    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd03bf0_4ec1_4000_9cc7_f44762360fe5.slice/crio-2f3a16211d9799c6d5d4e12066b52e1a3ab583a2a9236d8a3203b1c360f8be20 WatchSource:0}: Error finding container 2f3a16211d9799c6d5d4e12066b52e1a3ab583a2a9236d8a3203b1c360f8be20: Status 404 returned error can't find the container with id 2f3a16211d9799c6d5d4e12066b52e1a3ab583a2a9236d8a3203b1c360f8be20
Sep 17 23:42:20.224466 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:20.224417    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" event={"ID":"7bd03bf0-4ec1-4000-9cc7-f44762360fe5","Type":"ContainerStarted","Data":"16333ffa52444c822315babeb6d90869fe8a4b5de407510c865724e9fcef67d9"}
Sep 17 23:42:20.224868 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:20.224530    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" event={"ID":"7bd03bf0-4ec1-4000-9cc7-f44762360fe5","Type":"ContainerStarted","Data":"2f3a16211d9799c6d5d4e12066b52e1a3ab583a2a9236d8a3203b1c360f8be20"}
Sep 17 23:42:22.443122 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:22.443093    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"]
Sep 17 23:42:22.443556 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:22.443290    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" containerID="cri-o://a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4" gracePeriod=30
Sep 17 23:42:22.443556 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:22.443356    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kube-rbac-proxy" containerID="cri-o://ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03" gracePeriod=30
Sep 17 23:42:22.476419 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:22.476395    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:23.237976 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:23.237944    2571 generic.go:356] "Generic (PLEG): container finished" podID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerID="ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03" exitCode=2
Sep 17 23:42:23.238144 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:23.238025    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" event={"ID":"a5f6254d-d80a-4524-86e3-c3dac1960aa3","Type":"ContainerDied","Data":"ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03"}
Sep 17 23:42:24.243834 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:24.243802    2571 generic.go:356] "Generic (PLEG): container finished" podID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerID="16333ffa52444c822315babeb6d90869fe8a4b5de407510c865724e9fcef67d9" exitCode=0
Sep 17 23:42:24.244159 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:24.243836    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" event={"ID":"7bd03bf0-4ec1-4000-9cc7-f44762360fe5","Type":"ContainerDied","Data":"16333ffa52444c822315babeb6d90869fe8a4b5de407510c865724e9fcef67d9"}
Sep 17 23:42:24.245000 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:24.244985    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:42:24.571950 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:24.571866    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 23:42:24.576223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:24.576191    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 23:42:26.091810 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:26.091771    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:26.158385 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:26.158354    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:27.589121 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.589087    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:27.675860 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.675829    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:27.695934 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.695903    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"]
Sep 17 23:42:27.699593 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.699569    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.702279 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.702259    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 23:42:27.702554 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.702533    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 23:42:27.710661 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.710630    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"]
Sep 17 23:42:27.791524 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.791505    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:42:27.797180 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.797154    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:27.895964 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.895926    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kserve-provision-location\") pod \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") "
Sep 17 23:42:27.896067 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.895992    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-rbac-proxy-sar-config\") pod \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") "
Sep 17 23:42:27.896067 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896063    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls\") pod \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") "
Sep 17 23:42:27.896175 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896099    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mpjb\" (UniqueName: \"kubernetes.io/projected/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-api-access-5mpjb\") pod \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") "
Sep 17 23:42:27.896175 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896129    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-cabundle-cert\") pod \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\" (UID: \"a5f6254d-d80a-4524-86e3-c3dac1960aa3\") "
Sep 17 23:42:27.896340 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896233    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rnf\" (UniqueName: \"kubernetes.io/projected/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-api-access-k8rnf\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.896403 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896254    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5f6254d-d80a-4524-86e3-c3dac1960aa3" (UID: "a5f6254d-d80a-4524-86e3-c3dac1960aa3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:27.896403 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896260    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.896510 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896427    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.896602 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896461    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5f6254d-d80a-4524-86e3-c3dac1960aa3" (UID: "a5f6254d-d80a-4524-86e3-c3dac1960aa3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:42:27.896602 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896473    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.896731 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896636    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.896731 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896644    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5f6254d-d80a-4524-86e3-c3dac1960aa3" (UID: "a5f6254d-d80a-4524-86e3-c3dac1960aa3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:42:27.896826 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896737    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.896826 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896756    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.896826 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.896771    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.898865 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.898840    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5f6254d-d80a-4524-86e3-c3dac1960aa3" (UID: "a5f6254d-d80a-4524-86e3-c3dac1960aa3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:42:27.898865 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.898851    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-api-access-5mpjb" (OuterVolumeSpecName: "kube-api-access-5mpjb") pod "a5f6254d-d80a-4524-86e3-c3dac1960aa3" (UID: "a5f6254d-d80a-4524-86e3-c3dac1960aa3"). InnerVolumeSpecName "kube-api-access-5mpjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:42:27.946366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.946318    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:27.997919 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.997901    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rnf\" (UniqueName: \"kubernetes.io/projected/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-api-access-k8rnf\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.998025 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.997927    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.998025 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.997957    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.998025 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.997980    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.998173 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.998094    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.998173 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.998170    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mpjb\" (UniqueName: \"kubernetes.io/projected/a5f6254d-d80a-4524-86e3-c3dac1960aa3-kube-api-access-5mpjb\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.998278 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.998186    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5f6254d-d80a-4524-86e3-c3dac1960aa3-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:42:27.998360 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.998347    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.998671 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.998651    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:27.998762 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:27.998744    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:28.000230 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.000214    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-proxy-tls\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:28.007695 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.007680    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rnf\" (UniqueName: \"kubernetes.io/projected/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-api-access-k8rnf\") pod \"isvc-canary-rollback-predictor-7df6589c46-9cc4n\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:28.011497 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.011477    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:28.264027 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.263960    2571 generic.go:356] "Generic (PLEG): container finished" podID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerID="a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4" exitCode=0
Sep 17 23:42:28.264167 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.264082    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"
Sep 17 23:42:28.264167 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.264102    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" event={"ID":"a5f6254d-d80a-4524-86e3-c3dac1960aa3","Type":"ContainerDied","Data":"a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4"}
Sep 17 23:42:28.264273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.264180    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv" event={"ID":"a5f6254d-d80a-4524-86e3-c3dac1960aa3","Type":"ContainerDied","Data":"034c81f6cc72969126b24cb951b8d0b32f943d22d60948fc3dc78fa99ef28988"}
Sep 17 23:42:28.264273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.264212    2571 scope.go:117] "RemoveContainer" containerID="ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03"
Sep 17 23:42:28.266829 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.266804    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" event={"ID":"7bd03bf0-4ec1-4000-9cc7-f44762360fe5","Type":"ContainerStarted","Data":"0f62a0851e72ae4bb81d8add2b6842388049ffbd361f2e07dc7cc1d44c6002bf"}
Sep 17 23:42:28.266920 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.266869    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" event={"ID":"7bd03bf0-4ec1-4000-9cc7-f44762360fe5","Type":"ContainerStarted","Data":"613e56c0ea8d1ab7e05ad63811402a7f4b2f36eb3f5f1dcbf0fab2a918f83092"}
Sep 17 23:42:28.267186 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.267169    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:28.267260 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.267212    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:28.268452 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.268423    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 23:42:28.272737 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.272719    2571 scope.go:117] "RemoveContainer" containerID="a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4"
Sep 17 23:42:28.279716 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.279694    2571 scope.go:117] "RemoveContainer" containerID="5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d"
Sep 17 23:42:28.287185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.287163    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"]
Sep 17 23:42:28.287487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.287472    2571 scope.go:117] "RemoveContainer" containerID="ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03"
Sep 17 23:42:28.287780 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:42:28.287764    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03\": container with ID starting with ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03 not found: ID does not exist" containerID="ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03"
Sep 17 23:42:28.287836 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.287787    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03"} err="failed to get container status \"ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03\": rpc error: code = NotFound desc = could not find container \"ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03\": container with ID starting with ec359defe795b140893ee461b6644c230905a9178300d96c2dc54df854d88b03 not found: ID does not exist"
Sep 17 23:42:28.287836 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.287803    2571 scope.go:117] "RemoveContainer" containerID="a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4"
Sep 17 23:42:28.288064 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:42:28.288047    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4\": container with ID starting with a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4 not found: ID does not exist" containerID="a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4"
Sep 17 23:42:28.288108 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.288071    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4"} err="failed to get container status \"a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4\": rpc error: code = NotFound desc = could not find container \"a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4\": container with ID starting with a78f2a4174de54800fb2c10e6eb4d11fdd4ababdae05c6317aed6aa815ff6ef4 not found: ID does not exist"
Sep 17 23:42:28.288108 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.288087    2571 scope.go:117] "RemoveContainer" containerID="5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d"
Sep 17 23:42:28.288366 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:42:28.288348    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d\": container with ID starting with 5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d not found: ID does not exist" containerID="5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d"
Sep 17 23:42:28.288433 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.288374    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d"} err="failed to get container status \"5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d\": rpc error: code = NotFound desc = could not find container \"5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d\": container with ID starting with 5b8b7e3bca737264f6564df6e2050c7bbd811af9364b275ec2f7766adeccd40d not found: ID does not exist"
Sep 17 23:42:28.288719 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:42:28.288702    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bec4b8_253f_4c26_80f3_8f42a9c7c8d2.slice/crio-a2a649f9479a40250603e32ecadaa0a90cc927277ac9cea7337472d017a85a48 WatchSource:0}: Error finding container a2a649f9479a40250603e32ecadaa0a90cc927277ac9cea7337472d017a85a48: Status 404 returned error can't find the container with id a2a649f9479a40250603e32ecadaa0a90cc927277ac9cea7337472d017a85a48
Sep 17 23:42:28.291866 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.291832    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podStartSLOduration=2.337716888 podStartE2EDuration="10.291821482s" totalImagesPullingTime="3.539485016s" totalInitContainerRuntime="4.414619578s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:42:28.291400613 +0000 UTC m=+1872.752318343" watchObservedRunningTime="2026-09-17 23:42:28.291821482 +0000 UTC m=+1872.752739209"
Sep 17 23:42:28.308829 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.308810    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"]
Sep 17 23:42:28.314231 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:28.314213    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8555cb9565-kv5dv"]
Sep 17 23:42:29.272118 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:29.272076    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" event={"ID":"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2","Type":"ContainerStarted","Data":"1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5"}
Sep 17 23:42:29.272555 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:29.272154    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" event={"ID":"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2","Type":"ContainerStarted","Data":"a2a649f9479a40250603e32ecadaa0a90cc927277ac9cea7337472d017a85a48"}
Sep 17 23:42:29.273351 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:29.273302    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 23:42:30.058163 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:30.058133    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" path="/var/lib/kubelet/pods/a5f6254d-d80a-4524-86e3-c3dac1960aa3/volumes"
Sep 17 23:42:32.285697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:32.285667    2571 generic.go:356] "Generic (PLEG): container finished" podID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerID="1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5" exitCode=0
Sep 17 23:42:32.285977 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:32.285708    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" event={"ID":"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2","Type":"ContainerDied","Data":"1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5"}
Sep 17 23:42:33.291643 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:33.291613    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" event={"ID":"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2","Type":"ContainerStarted","Data":"fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449"}
Sep 17 23:42:33.292006 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:33.291669    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" event={"ID":"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2","Type":"ContainerStarted","Data":"a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab"}
Sep 17 23:42:33.292006 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:33.291921    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:33.292142 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:33.292080    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:33.293100 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:33.293076    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:42:33.310298 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:33.310257    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podStartSLOduration=2.495239542 podStartE2EDuration="6.310245679s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.815006137s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:42:33.309114984 +0000 UTC m=+1877.770032713" watchObservedRunningTime="2026-09-17 23:42:33.310245679 +0000 UTC m=+1877.771163409"
Sep 17 23:42:34.277520 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:34.277491    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:34.277930 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:34.277907    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 23:42:34.296046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:34.296019    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:42:35.300382 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:35.300348    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:42:40.304951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:40.304881    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:42:40.305547 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:40.305521    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:42:44.278830 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:44.278804    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:42:50.305992 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:50.305956    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:42:58.235657 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:58.235616    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"]
Sep 17 23:42:58.236073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:58.235852    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kserve-container" containerID="cri-o://613e56c0ea8d1ab7e05ad63811402a7f4b2f36eb3f5f1dcbf0fab2a918f83092" gracePeriod=30
Sep 17 23:42:58.236073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:58.235925    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy" containerID="cri-o://0f62a0851e72ae4bb81d8add2b6842388049ffbd361f2e07dc7cc1d44c6002bf" gracePeriod=30
Sep 17 23:42:58.281392 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:58.281362    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:42:58.394239 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:58.394216    2571 generic.go:356] "Generic (PLEG): container finished" podID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerID="0f62a0851e72ae4bb81d8add2b6842388049ffbd361f2e07dc7cc1d44c6002bf" exitCode=2
Sep 17 23:42:58.394351 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:58.394274    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" event={"ID":"7bd03bf0-4ec1-4000-9cc7-f44762360fe5","Type":"ContainerDied","Data":"0f62a0851e72ae4bb81d8add2b6842388049ffbd361f2e07dc7cc1d44c6002bf"}
Sep 17 23:42:59.274032 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:42:59.273999    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 23:43:00.306156 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:00.306121    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:43:03.415555 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.415525    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:03.446179 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446151    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"]
Sep 17 23:43:03.446486 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446474    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kube-rbac-proxy"
Sep 17 23:43:03.446550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446489    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kube-rbac-proxy"
Sep 17 23:43:03.446550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446500    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container"
Sep 17 23:43:03.446550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446505    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container"
Sep 17 23:43:03.446550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446519    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="storage-initializer"
Sep 17 23:43:03.446550 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446525    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="storage-initializer"
Sep 17 23:43:03.446697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446579    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kserve-container"
Sep 17 23:43:03.446697 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.446587    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f6254d-d80a-4524-86e3-c3dac1960aa3" containerName="kube-rbac-proxy"
Sep 17 23:43:03.449759 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.449745    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.452232 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.452211    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 23:43:03.452356 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.452247    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:43:03.462770 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.462749    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"]
Sep 17 23:43:03.563948 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.563926    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.564075 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.563970    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsw5x\" (UniqueName: \"kubernetes.io/projected/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-api-access-gsw5x\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.564075 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.563995    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c227356-0d9f-47c5-8f13-1b138d31a6c1-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.564075 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.564021    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.564075 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.564037    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.623960 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.623938    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:03.664963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.664940    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsw5x\" (UniqueName: \"kubernetes.io/projected/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-api-access-gsw5x\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.665072 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.664982    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c227356-0d9f-47c5-8f13-1b138d31a6c1-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.665072 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.665014    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.665072 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.665037    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.665230 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.665095    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.665523 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.665503    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.665671 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.665630    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.665723 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.665713    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.667578 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.667561    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c227356-0d9f-47c5-8f13-1b138d31a6c1-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.674498 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.674478    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsw5x\" (UniqueName: \"kubernetes.io/projected/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-api-access-gsw5x\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:03.759054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:03.759035    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:04.038779 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:04.038756    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"]
Sep 17 23:43:04.040519 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:43:04.040494    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c227356_0d9f_47c5_8f13_1b138d31a6c1.slice/crio-4815bce7de340e3e4eba7d8647f4329b6cf11259970348743168ca4a1d0cf990 WatchSource:0}: Error finding container 4815bce7de340e3e4eba7d8647f4329b6cf11259970348743168ca4a1d0cf990: Status 404 returned error can't find the container with id 4815bce7de340e3e4eba7d8647f4329b6cf11259970348743168ca4a1d0cf990
Sep 17 23:43:04.274053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:04.274015    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 23:43:04.418934 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:04.418879    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" event={"ID":"3c227356-0d9f-47c5-8f13-1b138d31a6c1","Type":"ContainerStarted","Data":"e78801ff5579b458fc49e0fc05d46ae0666892a7a40f76ecfb77d9101b56077f"}
Sep 17 23:43:04.419293 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:04.418998    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" event={"ID":"3c227356-0d9f-47c5-8f13-1b138d31a6c1","Type":"ContainerStarted","Data":"4815bce7de340e3e4eba7d8647f4329b6cf11259970348743168ca4a1d0cf990"}
Sep 17 23:43:09.274531 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:09.274452    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 23:43:09.274841 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:09.274575    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:43:09.441402 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:09.441367    2571 generic.go:356] "Generic (PLEG): container finished" podID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerID="e78801ff5579b458fc49e0fc05d46ae0666892a7a40f76ecfb77d9101b56077f" exitCode=0
Sep 17 23:43:09.441546 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:09.441442    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" event={"ID":"3c227356-0d9f-47c5-8f13-1b138d31a6c1","Type":"ContainerDied","Data":"e78801ff5579b458fc49e0fc05d46ae0666892a7a40f76ecfb77d9101b56077f"}
Sep 17 23:43:10.306082 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:10.306042    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:43:10.447924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:10.447895    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" event={"ID":"3c227356-0d9f-47c5-8f13-1b138d31a6c1","Type":"ContainerStarted","Data":"f9f471087b6dfe7de3c2e099e438ed1931bf14f6bec8a3e0199a058e2aaffd0d"}
Sep 17 23:43:10.448076 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:10.447955    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" event={"ID":"3c227356-0d9f-47c5-8f13-1b138d31a6c1","Type":"ContainerStarted","Data":"fa0e99af57217bb44b4ebc7f520a05c13b908b83bbd29dee4d0c9c7e084d5a32"}
Sep 17 23:43:10.448229 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:10.448207    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:10.448378 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:10.448350    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:10.449499 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:10.449476    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 23:43:10.467814 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:10.467777    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podStartSLOduration=2.699212106 podStartE2EDuration="7.467765842s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.768553736s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:43:10.465879029 +0000 UTC m=+1914.926796760" watchObservedRunningTime="2026-09-17 23:43:10.467765842 +0000 UTC m=+1914.928683571"
Sep 17 23:43:11.452140 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:11.452095    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 23:43:14.274282 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:14.274242    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 23:43:16.456721 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:16.456694    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:16.457382 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:16.457349    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 23:43:19.274416 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:19.274367    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 23:43:20.306291 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:20.306256    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:43:24.274503 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:24.274456    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 23:43:26.457736 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:26.457710    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:28.527115 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.527055    2571 generic.go:356] "Generic (PLEG): container finished" podID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerID="613e56c0ea8d1ab7e05ad63811402a7f4b2f36eb3f5f1dcbf0fab2a918f83092" exitCode=137
Sep 17 23:43:28.527437 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.527128    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" event={"ID":"7bd03bf0-4ec1-4000-9cc7-f44762360fe5","Type":"ContainerDied","Data":"613e56c0ea8d1ab7e05ad63811402a7f4b2f36eb3f5f1dcbf0fab2a918f83092"}
Sep 17 23:43:28.871373 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.871347    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:43:28.954315 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.954292    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-proxy-tls\") pod \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") "
Sep 17 23:43:28.954443 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.954356    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kserve-provision-location\") pod \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") "
Sep 17 23:43:28.954443 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.954380    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-rbac-proxy-sar-config\") pod \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") "
Sep 17 23:43:28.954443 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.954418    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24frx\" (UniqueName: \"kubernetes.io/projected/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-api-access-24frx\") pod \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") "
Sep 17 23:43:28.954572 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.954471    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-cabundle-cert\") pod \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\" (UID: \"7bd03bf0-4ec1-4000-9cc7-f44762360fe5\") "
Sep 17 23:43:28.954779 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.954757    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7bd03bf0-4ec1-4000-9cc7-f44762360fe5" (UID: "7bd03bf0-4ec1-4000-9cc7-f44762360fe5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:43:28.954855 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.954776    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7bd03bf0-4ec1-4000-9cc7-f44762360fe5" (UID: "7bd03bf0-4ec1-4000-9cc7-f44762360fe5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:43:28.956467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.956439    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-api-access-24frx" (OuterVolumeSpecName: "kube-api-access-24frx") pod "7bd03bf0-4ec1-4000-9cc7-f44762360fe5" (UID: "7bd03bf0-4ec1-4000-9cc7-f44762360fe5"). InnerVolumeSpecName "kube-api-access-24frx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:43:28.956567 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.956474    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7bd03bf0-4ec1-4000-9cc7-f44762360fe5" (UID: "7bd03bf0-4ec1-4000-9cc7-f44762360fe5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:43:28.964790 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:28.964769    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7bd03bf0-4ec1-4000-9cc7-f44762360fe5" (UID: "7bd03bf0-4ec1-4000-9cc7-f44762360fe5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:43:29.055119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.055070    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:43:29.055119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.055088    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24frx\" (UniqueName: \"kubernetes.io/projected/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kube-api-access-24frx\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:43:29.055119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.055099    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:43:29.055119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.055107    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:43:29.055119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.055114    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7bd03bf0-4ec1-4000-9cc7-f44762360fe5-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:43:29.533026 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.533007    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"
Sep 17 23:43:29.533389 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.533001    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8" event={"ID":"7bd03bf0-4ec1-4000-9cc7-f44762360fe5","Type":"ContainerDied","Data":"2f3a16211d9799c6d5d4e12066b52e1a3ab583a2a9236d8a3203b1c360f8be20"}
Sep 17 23:43:29.533389 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.533150    2571 scope.go:117] "RemoveContainer" containerID="0f62a0851e72ae4bb81d8add2b6842388049ffbd361f2e07dc7cc1d44c6002bf"
Sep 17 23:43:29.541706 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.541689    2571 scope.go:117] "RemoveContainer" containerID="613e56c0ea8d1ab7e05ad63811402a7f4b2f36eb3f5f1dcbf0fab2a918f83092"
Sep 17 23:43:29.548703 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.548682    2571 scope.go:117] "RemoveContainer" containerID="16333ffa52444c822315babeb6d90869fe8a4b5de407510c865724e9fcef67d9"
Sep 17 23:43:29.556381 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.556354    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"]
Sep 17 23:43:29.560741 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:29.560721    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-qw5s8"]
Sep 17 23:43:30.061846 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:30.061809    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" path="/var/lib/kubelet/pods/7bd03bf0-4ec1-4000-9cc7-f44762360fe5/volumes"
Sep 17 23:43:30.306011 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:30.305977    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:43:40.306292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:40.306265    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:43:42.439391 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:42.439352    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"]
Sep 17 23:43:42.439823 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:42.439587    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kserve-container" containerID="cri-o://fa0e99af57217bb44b4ebc7f520a05c13b908b83bbd29dee4d0c9c7e084d5a32" gracePeriod=30
Sep 17 23:43:42.439823 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:42.439621    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy" containerID="cri-o://f9f471087b6dfe7de3c2e099e438ed1931bf14f6bec8a3e0199a058e2aaffd0d" gracePeriod=30
Sep 17 23:43:42.462212 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:42.462190    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:42.585032 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:42.585011    2571 generic.go:356] "Generic (PLEG): container finished" podID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerID="f9f471087b6dfe7de3c2e099e438ed1931bf14f6bec8a3e0199a058e2aaffd0d" exitCode=2
Sep 17 23:43:42.585121 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:42.585082    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" event={"ID":"3c227356-0d9f-47c5-8f13-1b138d31a6c1","Type":"ContainerDied","Data":"f9f471087b6dfe7de3c2e099e438ed1931bf14f6bec8a3e0199a058e2aaffd0d"}
Sep 17 23:43:46.452446 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:46.452410    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 23:43:47.590168 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:47.590132    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:47.659068 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:47.659040    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:43:51.452300 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:51.452263    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 23:43:56.452524 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:56.452488    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 23:43:56.452871 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:56.452595    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:43:57.677529 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:43:57.677500    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:44:01.452877 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:01.452833    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 23:44:06.452936 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:06.452901    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 23:44:11.452673 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:11.452594    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Sep 17 23:44:12.702008 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:12.701949    2571 generic.go:356] "Generic (PLEG): container finished" podID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerID="fa0e99af57217bb44b4ebc7f520a05c13b908b83bbd29dee4d0c9c7e084d5a32" exitCode=137
Sep 17 23:44:12.702281 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:12.702019    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" event={"ID":"3c227356-0d9f-47c5-8f13-1b138d31a6c1","Type":"ContainerDied","Data":"fa0e99af57217bb44b4ebc7f520a05c13b908b83bbd29dee4d0c9c7e084d5a32"}
Sep 17 23:44:13.077482 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.077460    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:44:13.155179 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.155153    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsw5x\" (UniqueName: \"kubernetes.io/projected/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-api-access-gsw5x\") pod \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") "
Sep 17 23:44:13.155425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.155223    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kserve-provision-location\") pod \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") "
Sep 17 23:44:13.155425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.155310    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-cabundle-cert\") pod \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") "
Sep 17 23:44:13.155425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.155362    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-rbac-proxy-sar-config\") pod \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") "
Sep 17 23:44:13.155425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.155406    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c227356-0d9f-47c5-8f13-1b138d31a6c1-proxy-tls\") pod \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\" (UID: \"3c227356-0d9f-47c5-8f13-1b138d31a6c1\") "
Sep 17 23:44:13.155644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.155619    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c227356-0d9f-47c5-8f13-1b138d31a6c1" (UID: "3c227356-0d9f-47c5-8f13-1b138d31a6c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:13.155745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.155721    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c227356-0d9f-47c5-8f13-1b138d31a6c1" (UID: "3c227356-0d9f-47c5-8f13-1b138d31a6c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:13.157289 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.157257    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-api-access-gsw5x" (OuterVolumeSpecName: "kube-api-access-gsw5x") pod "3c227356-0d9f-47c5-8f13-1b138d31a6c1" (UID: "3c227356-0d9f-47c5-8f13-1b138d31a6c1"). InnerVolumeSpecName "kube-api-access-gsw5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:44:13.157590 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.157567    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c227356-0d9f-47c5-8f13-1b138d31a6c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c227356-0d9f-47c5-8f13-1b138d31a6c1" (UID: "3c227356-0d9f-47c5-8f13-1b138d31a6c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:13.164160 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.164139    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c227356-0d9f-47c5-8f13-1b138d31a6c1" (UID: "3c227356-0d9f-47c5-8f13-1b138d31a6c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:13.256807 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.256760    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsw5x\" (UniqueName: \"kubernetes.io/projected/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-api-access-gsw5x\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:44:13.256807 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.256780    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:44:13.256807 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.256790    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:44:13.256807 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.256798    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c227356-0d9f-47c5-8f13-1b138d31a6c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:44:13.256807 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.256806    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c227356-0d9f-47c5-8f13-1b138d31a6c1-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:44:13.708185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.708157    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd" event={"ID":"3c227356-0d9f-47c5-8f13-1b138d31a6c1","Type":"ContainerDied","Data":"4815bce7de340e3e4eba7d8647f4329b6cf11259970348743168ca4a1d0cf990"}
Sep 17 23:44:13.708185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.708182    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"
Sep 17 23:44:13.708579 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.708235    2571 scope.go:117] "RemoveContainer" containerID="f9f471087b6dfe7de3c2e099e438ed1931bf14f6bec8a3e0199a058e2aaffd0d"
Sep 17 23:44:13.716054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.716035    2571 scope.go:117] "RemoveContainer" containerID="fa0e99af57217bb44b4ebc7f520a05c13b908b83bbd29dee4d0c9c7e084d5a32"
Sep 17 23:44:13.723380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.723365    2571 scope.go:117] "RemoveContainer" containerID="e78801ff5579b458fc49e0fc05d46ae0666892a7a40f76ecfb77d9101b56077f"
Sep 17 23:44:13.729540 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.729520    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"]
Sep 17 23:44:13.735644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:13.735616    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-7r7cd"]
Sep 17 23:44:14.057157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:44:14.057093    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" path="/var/lib/kubelet/pods/3c227356-0d9f-47c5-8f13-1b138d31a6c1/volumes"
Sep 17 23:45:47.923020 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:47.922944    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"]
Sep 17 23:45:47.923776 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:47.923189    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" containerID="cri-o://a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab" gracePeriod=30
Sep 17 23:45:47.923776 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:47.923221    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kube-rbac-proxy" containerID="cri-o://fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449" gracePeriod=30
Sep 17 23:45:47.984937 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:47.984912    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:48.073952 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:48.073929    2571 generic.go:356] "Generic (PLEG): container finished" podID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerID="fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449" exitCode=2
Sep 17 23:45:48.074039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:48.074006    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" event={"ID":"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2","Type":"ContainerDied","Data":"fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449"}
Sep 17 23:45:50.300837 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:50.300799    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 23:45:50.305648 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:50.305626    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 23:45:51.451822 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.451801    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:45:51.533771 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.533708    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-proxy-tls\") pod \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") "
Sep 17 23:45:51.533771 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.533768    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8rnf\" (UniqueName: \"kubernetes.io/projected/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-api-access-k8rnf\") pod \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") "
Sep 17 23:45:51.533946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.533788    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-cabundle-cert\") pod \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") "
Sep 17 23:45:51.533946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.533810    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kserve-provision-location\") pod \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") "
Sep 17 23:45:51.533946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.533827    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-rbac-proxy-sar-config\") pod \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\" (UID: \"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2\") "
Sep 17 23:45:51.534157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.534137    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" (UID: "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:45:51.534157 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.534134    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" (UID: "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:51.534265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.534244    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" (UID: "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:45:51.535904 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.535885    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" (UID: "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:45:51.536009 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.535934    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-api-access-k8rnf" (OuterVolumeSpecName: "kube-api-access-k8rnf") pod "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" (UID: "48bec4b8-253f-4c26-80f3-8f42a9c7c8d2"). InnerVolumeSpecName "kube-api-access-k8rnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:45:51.634425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.634403    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:45:51.634425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.634421    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:45:51.634563 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.634431    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:45:51.634563 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.634440    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:45:51.634563 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:51.634449    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8rnf\" (UniqueName: \"kubernetes.io/projected/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2-kube-api-access-k8rnf\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:45:52.090845 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.090825    2571 generic.go:356] "Generic (PLEG): container finished" podID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerID="a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab" exitCode=0
Sep 17 23:45:52.090937 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.090868    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"
Sep 17 23:45:52.090937 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.090888    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" event={"ID":"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2","Type":"ContainerDied","Data":"a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab"}
Sep 17 23:45:52.090937 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.090933    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n" event={"ID":"48bec4b8-253f-4c26-80f3-8f42a9c7c8d2","Type":"ContainerDied","Data":"a2a649f9479a40250603e32ecadaa0a90cc927277ac9cea7337472d017a85a48"}
Sep 17 23:45:52.091050 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.090962    2571 scope.go:117] "RemoveContainer" containerID="fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449"
Sep 17 23:45:52.098890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.098874    2571 scope.go:117] "RemoveContainer" containerID="a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab"
Sep 17 23:45:52.105856 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.105838    2571 scope.go:117] "RemoveContainer" containerID="1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5"
Sep 17 23:45:52.112573 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.112552    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"]
Sep 17 23:45:52.113427 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.113412    2571 scope.go:117] "RemoveContainer" containerID="fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449"
Sep 17 23:45:52.113724 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:45:52.113707    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449\": container with ID starting with fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449 not found: ID does not exist" containerID="fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449"
Sep 17 23:45:52.113784 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.113731    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449"} err="failed to get container status \"fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449\": rpc error: code = NotFound desc = could not find container \"fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449\": container with ID starting with fb57d1a1eded22d6c484c7f4025b505bfceb044fef05e2a28d8c3a4f10ad8449 not found: ID does not exist"
Sep 17 23:45:52.113784 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.113746    2571 scope.go:117] "RemoveContainer" containerID="a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab"
Sep 17 23:45:52.113978 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:45:52.113959    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab\": container with ID starting with a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab not found: ID does not exist" containerID="a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab"
Sep 17 23:45:52.114045 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.113985    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab"} err="failed to get container status \"a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab\": rpc error: code = NotFound desc = could not find container \"a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab\": container with ID starting with a8d814dffdfbf1849edbcfc98c0efba08c78df828282c0c7caece66bffa786ab not found: ID does not exist"
Sep 17 23:45:52.114045 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.114006    2571 scope.go:117] "RemoveContainer" containerID="1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5"
Sep 17 23:45:52.114261 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:45:52.114243    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5\": container with ID starting with 1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5 not found: ID does not exist" containerID="1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5"
Sep 17 23:45:52.114303 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.114267    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5"} err="failed to get container status \"1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5\": rpc error: code = NotFound desc = could not find container \"1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5\": container with ID starting with 1ccc50ebfaafc853cd5df003c72350a688b272f0edfa6831b8c332fe6a95a2b5 not found: ID does not exist"
Sep 17 23:45:52.116974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.116957    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7df6589c46-9cc4n"]
Sep 17 23:45:52.947116 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.947091    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:52.984219 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984187    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"]
Sep 17 23:45:52.984562 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984546    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container"
Sep 17 23:45:52.984650 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984564    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container"
Sep 17 23:45:52.984650 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984580    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="storage-initializer"
Sep 17 23:45:52.984650 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984590    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="storage-initializer"
Sep 17 23:45:52.984650 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984608    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy"
Sep 17 23:45:52.984650 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984617    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy"
Sep 17 23:45:52.984650 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984629    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kube-rbac-proxy"
Sep 17 23:45:52.984650 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984637    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kube-rbac-proxy"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984656    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kserve-container"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984665    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kserve-container"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984679    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="storage-initializer"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984687    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="storage-initializer"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984696    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="storage-initializer"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984705    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="storage-initializer"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984714    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984722    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984733    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kserve-container"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984742    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kserve-container"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984815    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kserve-container"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984828    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kube-rbac-proxy"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984838    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c227356-0d9f-47c5-8f13-1b138d31a6c1" containerName="kube-rbac-proxy"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984847    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kserve-container"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984858    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd03bf0-4ec1-4000-9cc7-f44762360fe5" containerName="kserve-container"
Sep 17 23:45:52.985085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.984866    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" containerName="kube-rbac-proxy"
Sep 17 23:45:52.989568 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.989551    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"]
Sep 17 23:45:52.989675 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.989656    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:52.992045 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.992030    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 23:45:52.992432 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.992393    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:45:52.992553 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.992452    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:45:52.992553 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.992472    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:45:52.992722 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.992707    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:52.993595 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.993578    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 23:45:52.993746 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.993731    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:45:52.993816 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.993757    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:45:52.995000 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.994983    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 23:45:52.997998 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:52.997977    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"]
Sep 17 23:45:53.000686 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.000667    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"]
Sep 17 23:45:53.027835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.027814    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:53.044078 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044055    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.044152 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044082    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.044152 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044102    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.044152 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044125    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tccl2\" (UniqueName: \"kubernetes.io/projected/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-api-access-tccl2\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.044261 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044176    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6556a676-ef9c-4bc1-996d-dbe8514bb958-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.044261 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044220    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.044261 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044239    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnx5g\" (UniqueName: \"kubernetes.io/projected/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-api-access-cnx5g\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.044378 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044259    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.044378 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044275    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6556a676-ef9c-4bc1-996d-dbe8514bb958-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.044378 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.044294    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.144999 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.144976    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.145083 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145003    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.145083 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.145166 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145077    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tccl2\" (UniqueName: \"kubernetes.io/projected/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-api-access-tccl2\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.145166 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:45:53.145110    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 23:45:53.145166 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145117    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6556a676-ef9c-4bc1-996d-dbe8514bb958-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.145742 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145157    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.145742 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:45:53.145170    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls podName:a89601f3-a00b-4a36-bab6-ab756ba5ebae nodeName:}" failed. No retries permitted until 2026-09-17 23:45:53.645151839 +0000 UTC m=+2078.106069571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls") pod "isvc-canary-stop-predictor-596476db66-h77ln" (UID: "a89601f3-a00b-4a36-bab6-ab756ba5ebae") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 17 23:45:53.145894 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145773    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.145894 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145809    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnx5g\" (UniqueName: \"kubernetes.io/projected/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-api-access-cnx5g\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.145894 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145830    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.145894 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145847    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.145894 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145871    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6556a676-ef9c-4bc1-996d-dbe8514bb958-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.146153 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145904    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.146153 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.145903    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.146264 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.146148    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6556a676-ef9c-4bc1-996d-dbe8514bb958-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.146264 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.146226    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kserve-provision-location\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.146606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.146587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-cabundle-cert\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.147866 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.147847    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6556a676-ef9c-4bc1-996d-dbe8514bb958-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.155363 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.155339    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnx5g\" (UniqueName: \"kubernetes.io/projected/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-api-access-cnx5g\") pod \"isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.155464 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.155444    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tccl2\" (UniqueName: \"kubernetes.io/projected/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-api-access-tccl2\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.174741 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.174724    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:53.275525 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.275469    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:45:53.301502 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.301485    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:53.573201 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.573178    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"]
Sep 17 23:45:53.574957 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:45:53.574921    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6556a676_ef9c_4bc1_996d_dbe8514bb958.slice/crio-3acd98ffe1e901ba33739f975ee1b0351097e3bb75347db7e53475c161f3354c WatchSource:0}: Error finding container 3acd98ffe1e901ba33739f975ee1b0351097e3bb75347db7e53475c161f3354c: Status 404 returned error can't find the container with id 3acd98ffe1e901ba33739f975ee1b0351097e3bb75347db7e53475c161f3354c
Sep 17 23:45:53.649419 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.649392    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.651604 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.651586    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls\") pod \"isvc-canary-stop-predictor-596476db66-h77ln\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:53.908804 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:53.908775    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:54.058010 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:54.057982    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48bec4b8-253f-4c26-80f3-8f42a9c7c8d2" path="/var/lib/kubelet/pods/48bec4b8-253f-4c26-80f3-8f42a9c7c8d2/volumes"
Sep 17 23:45:54.100698 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:54.100670    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" event={"ID":"6556a676-ef9c-4bc1-996d-dbe8514bb958","Type":"ContainerStarted","Data":"eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa"}
Sep 17 23:45:54.100824 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:54.100717    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" event={"ID":"6556a676-ef9c-4bc1-996d-dbe8514bb958","Type":"ContainerStarted","Data":"3acd98ffe1e901ba33739f975ee1b0351097e3bb75347db7e53475c161f3354c"}
Sep 17 23:45:54.183219 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:54.183198    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"]
Sep 17 23:45:54.186084 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:45:54.186063    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89601f3_a00b_4a36_bab6_ab756ba5ebae.slice/crio-b33aef9d878f21fee39c57d80a2c30b5a6eaf15f77eab95b377a91b49c30cf05 WatchSource:0}: Error finding container b33aef9d878f21fee39c57d80a2c30b5a6eaf15f77eab95b377a91b49c30cf05: Status 404 returned error can't find the container with id b33aef9d878f21fee39c57d80a2c30b5a6eaf15f77eab95b377a91b49c30cf05
Sep 17 23:45:55.106515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:55.106470    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" event={"ID":"a89601f3-a00b-4a36-bab6-ab756ba5ebae","Type":"ContainerStarted","Data":"96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a"}
Sep 17 23:45:55.106850 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:55.106571    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" event={"ID":"a89601f3-a00b-4a36-bab6-ab756ba5ebae","Type":"ContainerStarted","Data":"b33aef9d878f21fee39c57d80a2c30b5a6eaf15f77eab95b377a91b49c30cf05"}
Sep 17 23:45:58.118984 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:58.118955    2571 generic.go:356] "Generic (PLEG): container finished" podID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerID="eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa" exitCode=0
Sep 17 23:45:58.119369 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:58.119032    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" event={"ID":"6556a676-ef9c-4bc1-996d-dbe8514bb958","Type":"ContainerDied","Data":"eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa"}
Sep 17 23:45:58.120633 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:58.120614    2571 generic.go:356] "Generic (PLEG): container finished" podID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerID="96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a" exitCode=0
Sep 17 23:45:58.120725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:58.120686    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" event={"ID":"a89601f3-a00b-4a36-bab6-ab756ba5ebae","Type":"ContainerDied","Data":"96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a"}
Sep 17 23:45:59.127053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.127022    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" event={"ID":"6556a676-ef9c-4bc1-996d-dbe8514bb958","Type":"ContainerStarted","Data":"aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86"}
Sep 17 23:45:59.127470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.127090    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" event={"ID":"6556a676-ef9c-4bc1-996d-dbe8514bb958","Type":"ContainerStarted","Data":"2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1"}
Sep 17 23:45:59.127470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.127304    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:45:59.129424 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.129401    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" event={"ID":"a89601f3-a00b-4a36-bab6-ab756ba5ebae","Type":"ContainerStarted","Data":"760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e"}
Sep 17 23:45:59.129535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.129447    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" event={"ID":"a89601f3-a00b-4a36-bab6-ab756ba5ebae","Type":"ContainerStarted","Data":"230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9"}
Sep 17 23:45:59.129747 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.129724    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:59.129857 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.129773    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:45:59.130848 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.130826    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:45:59.153369 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.153316    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podStartSLOduration=3.138356233 podStartE2EDuration="7.15330261s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.014946377s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:59.151544074 +0000 UTC m=+2083.612461804" watchObservedRunningTime="2026-09-17 23:45:59.15330261 +0000 UTC m=+2083.614220340"
Sep 17 23:45:59.176718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.176666    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podStartSLOduration=3.390635942 podStartE2EDuration="7.176654544s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786018602s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:59.174420208 +0000 UTC m=+2083.635337938" watchObservedRunningTime="2026-09-17 23:45:59.176654544 +0000 UTC m=+2083.637572349"
Sep 17 23:45:59.905023 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:45:59.904998    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:00.134465 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:00.134434    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:46:00.134465 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:00.134451    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:46:00.135846 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:00.135818    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:46:01.138929 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:01.138896    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:46:05.029874 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:05.029838    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:05.131092 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:05.131064    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:46:05.138423 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:05.138398    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:46:05.138979 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:05.138957    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:46:06.143398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:06.143371    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:46:06.143805 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:06.143780    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:46:15.138986 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:15.138952    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:46:16.144475 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:16.144440    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:46:25.139381 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:25.139319    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:46:26.143871 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:26.143837    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:46:35.139064 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:35.139026    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:46:36.144110 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:36.144075    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:46:37.954946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:37.954835    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:46:37.959472 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:37.957110    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:46:45.139642 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:45.139602    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:46:46.144604 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:46.144565    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:46:55.139309 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:55.139270    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:46:56.143802 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:46:56.143769    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:47:05.139473 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:05.139444    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:47:06.144266 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:06.144237    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:47:12.978868 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:12.978783    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"]
Sep 17 23:47:12.979259 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:12.979113    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" containerID="cri-o://2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1" gracePeriod=30
Sep 17 23:47:12.979259 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:12.979115    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kube-rbac-proxy" containerID="cri-o://aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86" gracePeriod=30
Sep 17 23:47:12.992892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:12.992868    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:13.012756 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:13.012734    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"]
Sep 17 23:47:13.012951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:13.012933    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" containerID="cri-o://230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9" gracePeriod=30
Sep 17 23:47:13.013062 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:13.013035    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kube-rbac-proxy" containerID="cri-o://760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e" gracePeriod=30
Sep 17 23:47:13.094450 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:13.094429    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:13.428738 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:13.428709    2571 generic.go:356] "Generic (PLEG): container finished" podID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerID="aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86" exitCode=2
Sep 17 23:47:13.428903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:13.428775    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" event={"ID":"6556a676-ef9c-4bc1-996d-dbe8514bb958","Type":"ContainerDied","Data":"aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86"}
Sep 17 23:47:13.431168 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:13.431143    2571 generic.go:356] "Generic (PLEG): container finished" podID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerID="760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e" exitCode=2
Sep 17 23:47:13.431273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:13.431181    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" event={"ID":"a89601f3-a00b-4a36-bab6-ab756ba5ebae","Type":"ContainerDied","Data":"760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e"}
Sep 17 23:47:15.134789 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:15.134749    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 23:47:15.141306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:15.141230    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 23:47:16.139188 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:16.139149    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Sep 17 23:47:16.144593 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:16.144567    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 23:47:17.070170 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.070148    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:47:17.108467 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.108446    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:47:17.146336 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146300    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6556a676-ef9c-4bc1-996d-dbe8514bb958-proxy-tls\") pod \"6556a676-ef9c-4bc1-996d-dbe8514bb958\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146374    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-cabundle-cert\") pod \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146412    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6556a676-ef9c-4bc1-996d-dbe8514bb958-kserve-provision-location\") pod \"6556a676-ef9c-4bc1-996d-dbe8514bb958\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146446    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls\") pod \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146473    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-cabundle-cert\") pod \"6556a676-ef9c-4bc1-996d-dbe8514bb958\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146517    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tccl2\" (UniqueName: \"kubernetes.io/projected/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-api-access-tccl2\") pod \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146542    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-rbac-proxy-sar-config\") pod \"6556a676-ef9c-4bc1-996d-dbe8514bb958\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146586    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kserve-provision-location\") pod \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146642    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cnx5g\" (UniqueName: \"kubernetes.io/projected/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-api-access-cnx5g\") pod \"6556a676-ef9c-4bc1-996d-dbe8514bb958\" (UID: \"6556a676-ef9c-4bc1-996d-dbe8514bb958\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146679    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-rbac-proxy-sar-config\") pod \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\" (UID: \"a89601f3-a00b-4a36-bab6-ab756ba5ebae\") "
Sep 17 23:47:17.146704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146695    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6556a676-ef9c-4bc1-996d-dbe8514bb958-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6556a676-ef9c-4bc1-996d-dbe8514bb958" (UID: "6556a676-ef9c-4bc1-996d-dbe8514bb958"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:47:17.147223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146809    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a89601f3-a00b-4a36-bab6-ab756ba5ebae" (UID: "a89601f3-a00b-4a36-bab6-ab756ba5ebae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:47:17.147223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146955    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a89601f3-a00b-4a36-bab6-ab756ba5ebae" (UID: "a89601f3-a00b-4a36-bab6-ab756ba5ebae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:47:17.147223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.147073    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a89601f3-a00b-4a36-bab6-ab756ba5ebae" (UID: "a89601f3-a00b-4a36-bab6-ab756ba5ebae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:47:17.147223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.147104    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6556a676-ef9c-4bc1-996d-dbe8514bb958" (UID: "6556a676-ef9c-4bc1-996d-dbe8514bb958"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:47:17.147223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.146993    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.147223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.147195    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6556a676-ef9c-4bc1-996d-dbe8514bb958-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.147708 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.147122    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6556a676-ef9c-4bc1-996d-dbe8514bb958" (UID: "6556a676-ef9c-4bc1-996d-dbe8514bb958"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:47:17.148967 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.148943    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-api-access-tccl2" (OuterVolumeSpecName: "kube-api-access-tccl2") pod "a89601f3-a00b-4a36-bab6-ab756ba5ebae" (UID: "a89601f3-a00b-4a36-bab6-ab756ba5ebae"). InnerVolumeSpecName "kube-api-access-tccl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:47:17.149051 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.148947    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-api-access-cnx5g" (OuterVolumeSpecName: "kube-api-access-cnx5g") pod "6556a676-ef9c-4bc1-996d-dbe8514bb958" (UID: "6556a676-ef9c-4bc1-996d-dbe8514bb958"). InnerVolumeSpecName "kube-api-access-cnx5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:47:17.149168 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.149145    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a89601f3-a00b-4a36-bab6-ab756ba5ebae" (UID: "a89601f3-a00b-4a36-bab6-ab756ba5ebae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:47:17.149168 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.149151    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6556a676-ef9c-4bc1-996d-dbe8514bb958-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6556a676-ef9c-4bc1-996d-dbe8514bb958" (UID: "6556a676-ef9c-4bc1-996d-dbe8514bb958"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:47:17.248164 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.248140    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a89601f3-a00b-4a36-bab6-ab756ba5ebae-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.248164 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.248163    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.248311 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.248176    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tccl2\" (UniqueName: \"kubernetes.io/projected/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-api-access-tccl2\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.248311 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.248186    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.248311 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.248195    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.248311 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.248202    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnx5g\" (UniqueName: \"kubernetes.io/projected/6556a676-ef9c-4bc1-996d-dbe8514bb958-kube-api-access-cnx5g\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.248311 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.248210    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a89601f3-a00b-4a36-bab6-ab756ba5ebae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.248311 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.248218    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6556a676-ef9c-4bc1-996d-dbe8514bb958-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:47:17.448821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.448756    2571 generic.go:356] "Generic (PLEG): container finished" podID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerID="2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1" exitCode=0
Sep 17 23:47:17.448821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.448812    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"
Sep 17 23:47:17.449002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.448827    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" event={"ID":"6556a676-ef9c-4bc1-996d-dbe8514bb958","Type":"ContainerDied","Data":"2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1"}
Sep 17 23:47:17.449002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.448884    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2" event={"ID":"6556a676-ef9c-4bc1-996d-dbe8514bb958","Type":"ContainerDied","Data":"3acd98ffe1e901ba33739f975ee1b0351097e3bb75347db7e53475c161f3354c"}
Sep 17 23:47:17.449002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.448921    2571 scope.go:117] "RemoveContainer" containerID="aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86"
Sep 17 23:47:17.451105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.451077    2571 generic.go:356] "Generic (PLEG): container finished" podID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerID="230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9" exitCode=0
Sep 17 23:47:17.451226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.451116    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" event={"ID":"a89601f3-a00b-4a36-bab6-ab756ba5ebae","Type":"ContainerDied","Data":"230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9"}
Sep 17 23:47:17.451226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.451127    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"
Sep 17 23:47:17.451226 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.451165    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln" event={"ID":"a89601f3-a00b-4a36-bab6-ab756ba5ebae","Type":"ContainerDied","Data":"b33aef9d878f21fee39c57d80a2c30b5a6eaf15f77eab95b377a91b49c30cf05"}
Sep 17 23:47:17.458290 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.458273    2571 scope.go:117] "RemoveContainer" containerID="2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1"
Sep 17 23:47:17.465616 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.465599    2571 scope.go:117] "RemoveContainer" containerID="eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa"
Sep 17 23:47:17.472539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.472518    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"]
Sep 17 23:47:17.473141 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.473129    2571 scope.go:117] "RemoveContainer" containerID="aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86"
Sep 17 23:47:17.473405 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:47:17.473386    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86\": container with ID starting with aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86 not found: ID does not exist" containerID="aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86"
Sep 17 23:47:17.473471 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.473411    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86"} err="failed to get container status \"aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86\": rpc error: code = NotFound desc = could not find container \"aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86\": container with ID starting with aca152d3c98d78738d0095d38dc564d8df8644f44b3e3219dcd7041af9542d86 not found: ID does not exist"
Sep 17 23:47:17.473471 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.473427    2571 scope.go:117] "RemoveContainer" containerID="2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1"
Sep 17 23:47:17.473652 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:47:17.473634    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1\": container with ID starting with 2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1 not found: ID does not exist" containerID="2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1"
Sep 17 23:47:17.473694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.473659    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1"} err="failed to get container status \"2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1\": rpc error: code = NotFound desc = could not find container \"2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1\": container with ID starting with 2b1252e41954014361931273cdcdcdb33a8107edbe5e7cc31f08213e2b8813a1 not found: ID does not exist"
Sep 17 23:47:17.473694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.473674    2571 scope.go:117] "RemoveContainer" containerID="eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa"
Sep 17 23:47:17.473881 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:47:17.473868    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa\": container with ID starting with eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa not found: ID does not exist" containerID="eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa"
Sep 17 23:47:17.473920 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.473885    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa"} err="failed to get container status \"eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa\": rpc error: code = NotFound desc = could not find container \"eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa\": container with ID starting with eb4437ed542c5b481cef65e46c2299cc489632b788563274d56f66e00e2151aa not found: ID does not exist"
Sep 17 23:47:17.473920 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.473896    2571 scope.go:117] "RemoveContainer" containerID="760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e"
Sep 17 23:47:17.478048 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.478028    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-75db96cc6c-pv7d2"]
Sep 17 23:47:17.481460 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.481434    2571 scope.go:117] "RemoveContainer" containerID="230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9"
Sep 17 23:47:17.488802 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.488778    2571 scope.go:117] "RemoveContainer" containerID="96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a"
Sep 17 23:47:17.489520 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.489502    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"]
Sep 17 23:47:17.496195 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.496176    2571 scope.go:117] "RemoveContainer" containerID="760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e"
Sep 17 23:47:17.496309 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.496277    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-596476db66-h77ln"]
Sep 17 23:47:17.496451 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:47:17.496434    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e\": container with ID starting with 760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e not found: ID does not exist" containerID="760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e"
Sep 17 23:47:17.496499 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.496457    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e"} err="failed to get container status \"760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e\": rpc error: code = NotFound desc = could not find container \"760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e\": container with ID starting with 760efee8683497f5f2fb7fd50dd97ac48de791466909314ba06d1313ace7ae3e not found: ID does not exist"
Sep 17 23:47:17.496499 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.496471    2571 scope.go:117] "RemoveContainer" containerID="230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9"
Sep 17 23:47:17.496693 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:47:17.496677    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9\": container with ID starting with 230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9 not found: ID does not exist" containerID="230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9"
Sep 17 23:47:17.496760 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.496700    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9"} err="failed to get container status \"230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9\": rpc error: code = NotFound desc = could not find container \"230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9\": container with ID starting with 230f73d78ee9bfd5e7a05dd18bfd9cf979dd14cd3c145154094f6f24f1ef59e9 not found: ID does not exist"
Sep 17 23:47:17.496760 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.496719    2571 scope.go:117] "RemoveContainer" containerID="96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a"
Sep 17 23:47:17.496965 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:47:17.496950    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a\": container with ID starting with 96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a not found: ID does not exist" containerID="96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a"
Sep 17 23:47:17.497002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:17.496968    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a"} err="failed to get container status \"96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a\": rpc error: code = NotFound desc = could not find container \"96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a\": container with ID starting with 96c569fd828d3dc547729684dfc62547a31a3b22b47f7e54f5bbc23361948b9a not found: ID does not exist"
Sep 17 23:47:18.057797 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:18.057768    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" path="/var/lib/kubelet/pods/6556a676-ef9c-4bc1-996d-dbe8514bb958/volumes"
Sep 17 23:47:18.058259 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:18.058240    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" path="/var/lib/kubelet/pods/a89601f3-a00b-4a36-bab6-ab756ba5ebae/volumes"
Sep 17 23:47:23.220247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.220211    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:23.235896 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.235864    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"]
Sep 17 23:47:23.236130 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236119    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="storage-initializer"
Sep 17 23:47:23.236172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236132    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="storage-initializer"
Sep 17 23:47:23.236172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236141    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kube-rbac-proxy"
Sep 17 23:47:23.236172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236147    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kube-rbac-proxy"
Sep 17 23:47:23.236172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236155    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container"
Sep 17 23:47:23.236172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236160    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container"
Sep 17 23:47:23.236172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236167    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="storage-initializer"
Sep 17 23:47:23.236172 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236172    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="storage-initializer"
Sep 17 23:47:23.236398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236182    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kube-rbac-proxy"
Sep 17 23:47:23.236398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236187    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kube-rbac-proxy"
Sep 17 23:47:23.236398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236198    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container"
Sep 17 23:47:23.236398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236202    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container"
Sep 17 23:47:23.236398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236247    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kube-rbac-proxy"
Sep 17 23:47:23.236398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236255    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89601f3-a00b-4a36-bab6-ab756ba5ebae" containerName="kserve-container"
Sep 17 23:47:23.236398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236262    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kserve-container"
Sep 17 23:47:23.236398 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.236269    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6556a676-ef9c-4bc1-996d-dbe8514bb958" containerName="kube-rbac-proxy"
Sep 17 23:47:23.239191 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.239177    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.241863 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.241842    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:47:23.242123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.242098    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:47:23.242123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.242109    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 17 23:47:23.242395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.242378    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:47:23.242452 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.242396    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:47:23.242452 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.242378    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:47:23.243114 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.243098    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 17 23:47:23.248970 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.248948    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"]
Sep 17 23:47:23.288757 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.288736    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6512f069-a146-4131-8f67-421b6cfa2d28-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.288847 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.288771    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.288847 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.288789    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bng6v\" (UniqueName: \"kubernetes.io/projected/6512f069-a146-4131-8f67-421b6cfa2d28-kube-api-access-bng6v\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.288928 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.288847    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.288928 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.288901    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6512f069-a146-4131-8f67-421b6cfa2d28-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.390047 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.390022    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.390150 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.390049    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bng6v\" (UniqueName: \"kubernetes.io/projected/6512f069-a146-4131-8f67-421b6cfa2d28-kube-api-access-bng6v\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.390150 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.390071    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.390150 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.390131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6512f069-a146-4131-8f67-421b6cfa2d28-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.390308 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.390156    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6512f069-a146-4131-8f67-421b6cfa2d28-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.390542 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.390526    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6512f069-a146-4131-8f67-421b6cfa2d28-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.390764 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.390747    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.390812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.390762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.392570 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.392553    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6512f069-a146-4131-8f67-421b6cfa2d28-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.394180 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.394162    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:23.398818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.398798    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bng6v\" (UniqueName: \"kubernetes.io/projected/6512f069-a146-4131-8f67-421b6cfa2d28-kube-api-access-bng6v\") pod \"isvc-lightgbm-predictor-54f85ddf88-5gp66\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.550235 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.550174    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:23.821235 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:23.821211    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"]
Sep 17 23:47:23.823100 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:47:23.823075    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6512f069_a146_4131_8f67_421b6cfa2d28.slice/crio-76e4b8a010b6c524a311ff24208d1129aef12c4ce625bc0c922d494b41b22039 WatchSource:0}: Error finding container 76e4b8a010b6c524a311ff24208d1129aef12c4ce625bc0c922d494b41b22039: Status 404 returned error can't find the container with id 76e4b8a010b6c524a311ff24208d1129aef12c4ce625bc0c922d494b41b22039
Sep 17 23:47:24.479217 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:24.479180    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" event={"ID":"6512f069-a146-4131-8f67-421b6cfa2d28","Type":"ContainerStarted","Data":"b6da171acc3a32b70e65081399da138bedb130b3e838ecb590257e04c2f6de7e"}
Sep 17 23:47:24.479597 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:24.479255    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" event={"ID":"6512f069-a146-4131-8f67-421b6cfa2d28","Type":"ContainerStarted","Data":"76e4b8a010b6c524a311ff24208d1129aef12c4ce625bc0c922d494b41b22039"}
Sep 17 23:47:28.495925 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:28.495890    2571 generic.go:356] "Generic (PLEG): container finished" podID="6512f069-a146-4131-8f67-421b6cfa2d28" containerID="b6da171acc3a32b70e65081399da138bedb130b3e838ecb590257e04c2f6de7e" exitCode=0
Sep 17 23:47:28.496292 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:28.495942    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" event={"ID":"6512f069-a146-4131-8f67-421b6cfa2d28","Type":"ContainerDied","Data":"b6da171acc3a32b70e65081399da138bedb130b3e838ecb590257e04c2f6de7e"}
Sep 17 23:47:28.497072 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:28.497055    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:47:35.529730 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:35.529700    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" event={"ID":"6512f069-a146-4131-8f67-421b6cfa2d28","Type":"ContainerStarted","Data":"d26b55f14994b690f3741160eca3d889fcae8aa3d93a693b3bdbab641f9bf91c"}
Sep 17 23:47:35.530217 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:35.529763    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" event={"ID":"6512f069-a146-4131-8f67-421b6cfa2d28","Type":"ContainerStarted","Data":"156ec8cdfb79f210fd109a73355bbdf0d52717606fedcf2d5da7a296499598cf"}
Sep 17 23:47:35.530217 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:35.529948    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:35.549221 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:35.549175    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podStartSLOduration=2.291762932 podStartE2EDuration="12.549162633s" totalImagesPullingTime="6.365233396s" totalInitContainerRuntime="3.892166305s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:47:35.546942393 +0000 UTC m=+2180.007860122" watchObservedRunningTime="2026-09-17 23:47:35.549162633 +0000 UTC m=+2180.010080362"
Sep 17 23:47:36.534046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:36.534019    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:36.535344 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:36.535294    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:47:37.537885 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:37.537852    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:47:42.541668 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:42.541637    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:47:42.542111 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:42.542084    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:47:45.622293 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:45.622261    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:50.770512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.770467    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:50.784950 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.784921    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"]
Sep 17 23:47:50.788538 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.788523    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.791297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.791277    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 23:47:50.791425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.791285    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 23:47:50.797946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.797922    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"]
Sep 17 23:47:50.895619 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.895593    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.895723 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.895626    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrx9\" (UniqueName: \"kubernetes.io/projected/02c08576-2c46-4c6b-8734-215af02e5c26-kube-api-access-qzrx9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.895723 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.895658    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c08576-2c46-4c6b-8734-215af02e5c26-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.895794 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.895728    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.895794 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.895762    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.997000 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.996970    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c08576-2c46-4c6b-8734-215af02e5c26-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.997109 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.997017    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.997109 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.997036    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.997109 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.997062    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.997109 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.997082    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrx9\" (UniqueName: \"kubernetes.io/projected/02c08576-2c46-4c6b-8734-215af02e5c26-kube-api-access-qzrx9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.997309 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:47:50.997130    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 23:47:50.997309 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:47:50.997199    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls podName:02c08576-2c46-4c6b-8734-215af02e5c26 nodeName:}" failed. No retries permitted until 2026-09-17 23:47:51.497180343 +0000 UTC m=+2195.958098055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" (UID: "02c08576-2c46-4c6b-8734-215af02e5c26") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 23:47:50.997463 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.997394    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c08576-2c46-4c6b-8734-215af02e5c26-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.997733 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.997712    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:50.997733 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:50.997723    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:51.006073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:51.006053    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrx9\" (UniqueName: \"kubernetes.io/projected/02c08576-2c46-4c6b-8734-215af02e5c26-kube-api-access-qzrx9\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:51.348301 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:51.348276    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:47:51.500487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:51.500446    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:51.502903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:51.502887    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:51.699889 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:51.699828    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:51.970951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:51.970925    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"]
Sep 17 23:47:51.972645 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:47:51.972617    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c08576_2c46_4c6b_8734_215af02e5c26.slice/crio-ee636d537bbf45fe208475ab29e9fe966d26220c1f7fdb8d8c0d6ee1d7b852ae WatchSource:0}: Error finding container ee636d537bbf45fe208475ab29e9fe966d26220c1f7fdb8d8c0d6ee1d7b852ae: Status 404 returned error can't find the container with id ee636d537bbf45fe208475ab29e9fe966d26220c1f7fdb8d8c0d6ee1d7b852ae
Sep 17 23:47:52.542770 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:52.542732    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:47:52.597285 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:52.597252    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" event={"ID":"02c08576-2c46-4c6b-8734-215af02e5c26","Type":"ContainerStarted","Data":"36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27"}
Sep 17 23:47:52.597464 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:52.597316    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" event={"ID":"02c08576-2c46-4c6b-8734-215af02e5c26","Type":"ContainerStarted","Data":"ee636d537bbf45fe208475ab29e9fe966d26220c1f7fdb8d8c0d6ee1d7b852ae"}
Sep 17 23:47:56.612665 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:56.612628    2571 generic.go:356] "Generic (PLEG): container finished" podID="02c08576-2c46-4c6b-8734-215af02e5c26" containerID="36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27" exitCode=0
Sep 17 23:47:56.613008 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:56.612703    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" event={"ID":"02c08576-2c46-4c6b-8734-215af02e5c26","Type":"ContainerDied","Data":"36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27"}
Sep 17 23:47:57.619166 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:57.619132    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" event={"ID":"02c08576-2c46-4c6b-8734-215af02e5c26","Type":"ContainerStarted","Data":"d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91"}
Sep 17 23:47:57.619636 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:57.619201    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" event={"ID":"02c08576-2c46-4c6b-8734-215af02e5c26","Type":"ContainerStarted","Data":"f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d"}
Sep 17 23:47:57.619636 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:57.619432    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:47:57.645692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:57.645641    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" podStartSLOduration=3.726320922 podStartE2EDuration="7.645626293s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.919305371s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:47:57.645008131 +0000 UTC m=+2202.105925862" watchObservedRunningTime="2026-09-17 23:47:57.645626293 +0000 UTC m=+2202.106544024"
Sep 17 23:47:58.623315 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:47:58.623291    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:48:02.542578 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:02.542541    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:48:04.632530 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:04.632503    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:48:12.543113 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:12.543068    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:48:22.542789 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:22.542751    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:48:32.542812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:32.542774    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:48:34.636616 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:34.636588    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:48:40.837169 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:40.837081    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"]
Sep 17 23:48:40.837553 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:40.837426    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kserve-container" containerID="cri-o://f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d" gracePeriod=30
Sep 17 23:48:40.837553 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:40.837479    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kube-rbac-proxy" containerID="cri-o://d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91" gracePeriod=30
Sep 17 23:48:40.850538 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:40.850507    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:48:41.801056 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:41.801026    2571 generic.go:356] "Generic (PLEG): container finished" podID="02c08576-2c46-4c6b-8734-215af02e5c26" containerID="d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91" exitCode=2
Sep 17 23:48:41.801216 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:41.801065    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" event={"ID":"02c08576-2c46-4c6b-8734-215af02e5c26","Type":"ContainerDied","Data":"d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91"}
Sep 17 23:48:42.542906 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:42.542863    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:48:43.053581 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:43.053542    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:48:44.628193 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:44.628158    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Sep 17 23:48:44.633652 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:44.633615    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.46:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.133.0.46:8080: connect: connection refused"
Sep 17 23:48:45.775847 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.775824    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:48:45.819611 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.819580    2571 generic.go:356] "Generic (PLEG): container finished" podID="02c08576-2c46-4c6b-8734-215af02e5c26" containerID="f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d" exitCode=0
Sep 17 23:48:45.819711 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.819628    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" event={"ID":"02c08576-2c46-4c6b-8734-215af02e5c26","Type":"ContainerDied","Data":"f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d"}
Sep 17 23:48:45.819711 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.819677    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq" event={"ID":"02c08576-2c46-4c6b-8734-215af02e5c26","Type":"ContainerDied","Data":"ee636d537bbf45fe208475ab29e9fe966d26220c1f7fdb8d8c0d6ee1d7b852ae"}
Sep 17 23:48:45.819711 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.819632    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"
Sep 17 23:48:45.819821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.819706    2571 scope.go:117] "RemoveContainer" containerID="d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91"
Sep 17 23:48:45.827880 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.827859    2571 scope.go:117] "RemoveContainer" containerID="f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d"
Sep 17 23:48:45.835192 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.835175    2571 scope.go:117] "RemoveContainer" containerID="36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27"
Sep 17 23:48:45.842198 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.842169    2571 scope.go:117] "RemoveContainer" containerID="d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91"
Sep 17 23:48:45.842470 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:48:45.842452    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91\": container with ID starting with d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91 not found: ID does not exist" containerID="d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91"
Sep 17 23:48:45.842530 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.842482    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91"} err="failed to get container status \"d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91\": rpc error: code = NotFound desc = could not find container \"d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91\": container with ID starting with d1c543a6bfcc504140cfdb08fefcfc3fc1814bc93e0d494704cc0713a1a43d91 not found: ID does not exist"
Sep 17 23:48:45.842530 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.842498    2571 scope.go:117] "RemoveContainer" containerID="f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d"
Sep 17 23:48:45.842751 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:48:45.842733    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d\": container with ID starting with f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d not found: ID does not exist" containerID="f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d"
Sep 17 23:48:45.842801 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.842760    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d"} err="failed to get container status \"f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d\": rpc error: code = NotFound desc = could not find container \"f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d\": container with ID starting with f1bdd1d9ad2e979e0c0d79efc2e826c8558d1a8d5e474cc170a571200d95884d not found: ID does not exist"
Sep 17 23:48:45.842801 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.842774    2571 scope.go:117] "RemoveContainer" containerID="36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27"
Sep 17 23:48:45.842982 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:48:45.842968    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27\": container with ID starting with 36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27 not found: ID does not exist" containerID="36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27"
Sep 17 23:48:45.843020 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.842987    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27"} err="failed to get container status \"36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27\": rpc error: code = NotFound desc = could not find container \"36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27\": container with ID starting with 36ebfee49d2b4926431fcdc69aabb2c85368d9e413696734ee1b23bdf09f5a27 not found: ID does not exist"
Sep 17 23:48:45.904643 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.904592    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-cabundle-cert\") pod \"02c08576-2c46-4c6b-8734-215af02e5c26\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") "
Sep 17 23:48:45.904643 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.904631    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-kube-rbac-proxy-sar-config\") pod \"02c08576-2c46-4c6b-8734-215af02e5c26\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") "
Sep 17 23:48:45.904812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.904666    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzrx9\" (UniqueName: \"kubernetes.io/projected/02c08576-2c46-4c6b-8734-215af02e5c26-kube-api-access-qzrx9\") pod \"02c08576-2c46-4c6b-8734-215af02e5c26\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") "
Sep 17 23:48:45.904812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.904704    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c08576-2c46-4c6b-8734-215af02e5c26-kserve-provision-location\") pod \"02c08576-2c46-4c6b-8734-215af02e5c26\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") "
Sep 17 23:48:45.904812 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.904720    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls\") pod \"02c08576-2c46-4c6b-8734-215af02e5c26\" (UID: \"02c08576-2c46-4c6b-8734-215af02e5c26\") "
Sep 17 23:48:45.904991 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.904965    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02c08576-2c46-4c6b-8734-215af02e5c26" (UID: "02c08576-2c46-4c6b-8734-215af02e5c26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:48:45.904991 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.904980    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02c08576-2c46-4c6b-8734-215af02e5c26" (UID: "02c08576-2c46-4c6b-8734-215af02e5c26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:48:45.905093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.905001    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c08576-2c46-4c6b-8734-215af02e5c26-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02c08576-2c46-4c6b-8734-215af02e5c26" (UID: "02c08576-2c46-4c6b-8734-215af02e5c26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:48:45.906864 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.906838    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c08576-2c46-4c6b-8734-215af02e5c26-kube-api-access-qzrx9" (OuterVolumeSpecName: "kube-api-access-qzrx9") pod "02c08576-2c46-4c6b-8734-215af02e5c26" (UID: "02c08576-2c46-4c6b-8734-215af02e5c26"). InnerVolumeSpecName "kube-api-access-qzrx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:48:45.906864 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:45.906840    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02c08576-2c46-4c6b-8734-215af02e5c26" (UID: "02c08576-2c46-4c6b-8734-215af02e5c26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:48:46.005493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.005467    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzrx9\" (UniqueName: \"kubernetes.io/projected/02c08576-2c46-4c6b-8734-215af02e5c26-kube-api-access-qzrx9\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:48:46.005493 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.005487    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02c08576-2c46-4c6b-8734-215af02e5c26-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:48:46.005631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.005496    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02c08576-2c46-4c6b-8734-215af02e5c26-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:48:46.005631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.005506    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:48:46.005631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.005515    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02c08576-2c46-4c6b-8734-215af02e5c26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:48:46.017239 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.017218    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:48:46.044010 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.043981    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"]
Sep 17 23:48:46.044280 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.044269    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kserve-container"
Sep 17 23:48:46.044336 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.044282    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kserve-container"
Sep 17 23:48:46.044336 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.044297    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="storage-initializer"
Sep 17 23:48:46.044336 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.044303    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="storage-initializer"
Sep 17 23:48:46.044336 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.044308    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kube-rbac-proxy"
Sep 17 23:48:46.044336 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.044313    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kube-rbac-proxy"
Sep 17 23:48:46.044490 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.044378    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kserve-container"
Sep 17 23:48:46.044490 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.044386    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" containerName="kube-rbac-proxy"
Sep 17 23:48:46.048944 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.048927    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.051681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.051662    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 23:48:46.051785 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.051699    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 23:48:46.059163 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.059144    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"]
Sep 17 23:48:46.106610 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.106592    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.106708 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.106625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.106758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.106708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.106758 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.106741    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxdp4\" (UniqueName: \"kubernetes.io/projected/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-api-access-qxdp4\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.106836 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.106767    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.138989 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.138964    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"]
Sep 17 23:48:46.141366 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.141346    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-d4bhq"]
Sep 17 23:48:46.180044 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.179969    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:48:46.208115 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.208094    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.208174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.208130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.208174 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.208165    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.208245 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.208184    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxdp4\" (UniqueName: \"kubernetes.io/projected/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-api-access-qxdp4\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.208304 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.208280    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.208695 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.208671    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.208859 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.208844    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.208906 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.208846    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.210744 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.210730    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.217767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.217748    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxdp4\" (UniqueName: \"kubernetes.io/projected/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-api-access-qxdp4\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-7t82l\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.359718 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.359699    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:46.636502 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.636471    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"]
Sep 17 23:48:46.636814 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:48:46.636790    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fe087d_ba6a_4a1f_9cbb_6c3aed120413.slice/crio-f626cc059238d25e1c666ebc7134ad6dd763e979bd86067b5039611bf4f08f88 WatchSource:0}: Error finding container f626cc059238d25e1c666ebc7134ad6dd763e979bd86067b5039611bf4f08f88: Status 404 returned error can't find the container with id f626cc059238d25e1c666ebc7134ad6dd763e979bd86067b5039611bf4f08f88
Sep 17 23:48:46.826282 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.826250    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" event={"ID":"67fe087d-ba6a-4a1f-9cbb-6c3aed120413","Type":"ContainerStarted","Data":"738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46"}
Sep 17 23:48:46.826752 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:46.826312    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" event={"ID":"67fe087d-ba6a-4a1f-9cbb-6c3aed120413","Type":"ContainerStarted","Data":"f626cc059238d25e1c666ebc7134ad6dd763e979bd86067b5039611bf4f08f88"}
Sep 17 23:48:48.058353 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:48.058293    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c08576-2c46-4c6b-8734-215af02e5c26" path="/var/lib/kubelet/pods/02c08576-2c46-4c6b-8734-215af02e5c26/volumes"
Sep 17 23:48:50.849351 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:50.849303    2571 generic.go:356] "Generic (PLEG): container finished" podID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerID="738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46" exitCode=0
Sep 17 23:48:50.849666 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:50.849361    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" event={"ID":"67fe087d-ba6a-4a1f-9cbb-6c3aed120413","Type":"ContainerDied","Data":"738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46"}
Sep 17 23:48:51.855511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:51.855477    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" event={"ID":"67fe087d-ba6a-4a1f-9cbb-6c3aed120413","Type":"ContainerStarted","Data":"344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e"}
Sep 17 23:48:51.855916 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:51.855561    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" event={"ID":"67fe087d-ba6a-4a1f-9cbb-6c3aed120413","Type":"ContainerStarted","Data":"ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026"}
Sep 17 23:48:51.855916 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:51.855759    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:51.855916 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:51.855801    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:48:51.879378 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:51.879312    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" podStartSLOduration=1.7726254890000002 podStartE2EDuration="5.879300471s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.106674982s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:48:51.876692817 +0000 UTC m=+2256.337610550" watchObservedRunningTime="2026-09-17 23:48:51.879300471 +0000 UTC m=+2256.340218201"
Sep 17 23:48:53.054053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:53.054017    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Sep 17 23:48:57.865498 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:48:57.865469    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:49:03.054109 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:03.054079    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:49:04.038040 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:04.038004    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"]
Sep 17 23:49:04.038310 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:04.038286    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container" containerID="cri-o://156ec8cdfb79f210fd109a73355bbdf0d52717606fedcf2d5da7a296499598cf" gracePeriod=30
Sep 17 23:49:04.038435 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:04.038387    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kube-rbac-proxy" containerID="cri-o://d26b55f14994b690f3741160eca3d889fcae8aa3d93a693b3bdbab641f9bf91c" gracePeriod=30
Sep 17 23:49:04.064967 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:04.064946    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:49:04.908149 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:04.908119    2571 generic.go:356] "Generic (PLEG): container finished" podID="6512f069-a146-4131-8f67-421b6cfa2d28" containerID="d26b55f14994b690f3741160eca3d889fcae8aa3d93a693b3bdbab641f9bf91c" exitCode=2
Sep 17 23:49:04.908301 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:04.908195    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" event={"ID":"6512f069-a146-4131-8f67-421b6cfa2d28","Type":"ContainerDied","Data":"d26b55f14994b690f3741160eca3d889fcae8aa3d93a693b3bdbab641f9bf91c"}
Sep 17 23:49:07.538981 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:07.538944    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Sep 17 23:49:09.208628 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:09.208594    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:49:09.631256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:09.631225    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:49:09.929949 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:09.929882    2571 generic.go:356] "Generic (PLEG): container finished" podID="6512f069-a146-4131-8f67-421b6cfa2d28" containerID="156ec8cdfb79f210fd109a73355bbdf0d52717606fedcf2d5da7a296499598cf" exitCode=0
Sep 17 23:49:09.930073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:09.929951    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" event={"ID":"6512f069-a146-4131-8f67-421b6cfa2d28","Type":"ContainerDied","Data":"156ec8cdfb79f210fd109a73355bbdf0d52717606fedcf2d5da7a296499598cf"}
Sep 17 23:49:10.072016 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.071994    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:49:10.180256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.180183    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bng6v\" (UniqueName: \"kubernetes.io/projected/6512f069-a146-4131-8f67-421b6cfa2d28-kube-api-access-bng6v\") pod \"6512f069-a146-4131-8f67-421b6cfa2d28\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") "
Sep 17 23:49:10.180256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.180243    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6512f069-a146-4131-8f67-421b6cfa2d28-kserve-provision-location\") pod \"6512f069-a146-4131-8f67-421b6cfa2d28\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") "
Sep 17 23:49:10.180499 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.180282    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6512f069-a146-4131-8f67-421b6cfa2d28-proxy-tls\") pod \"6512f069-a146-4131-8f67-421b6cfa2d28\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") "
Sep 17 23:49:10.180499 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.180304    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-kube-rbac-proxy-sar-config\") pod \"6512f069-a146-4131-8f67-421b6cfa2d28\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") "
Sep 17 23:49:10.180499 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.180347    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-cabundle-cert\") pod \"6512f069-a146-4131-8f67-421b6cfa2d28\" (UID: \"6512f069-a146-4131-8f67-421b6cfa2d28\") "
Sep 17 23:49:10.180691 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.180663    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6512f069-a146-4131-8f67-421b6cfa2d28" (UID: "6512f069-a146-4131-8f67-421b6cfa2d28"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:49:10.180691 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.180661    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6512f069-a146-4131-8f67-421b6cfa2d28-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6512f069-a146-4131-8f67-421b6cfa2d28" (UID: "6512f069-a146-4131-8f67-421b6cfa2d28"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:49:10.180835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.180737    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6512f069-a146-4131-8f67-421b6cfa2d28" (UID: "6512f069-a146-4131-8f67-421b6cfa2d28"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:49:10.182524 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.182500    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6512f069-a146-4131-8f67-421b6cfa2d28-kube-api-access-bng6v" (OuterVolumeSpecName: "kube-api-access-bng6v") pod "6512f069-a146-4131-8f67-421b6cfa2d28" (UID: "6512f069-a146-4131-8f67-421b6cfa2d28"). InnerVolumeSpecName "kube-api-access-bng6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:49:10.182607 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.182593    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6512f069-a146-4131-8f67-421b6cfa2d28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6512f069-a146-4131-8f67-421b6cfa2d28" (UID: "6512f069-a146-4131-8f67-421b6cfa2d28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:49:10.281585 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.281562    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6512f069-a146-4131-8f67-421b6cfa2d28-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:10.281585 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.281583    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6512f069-a146-4131-8f67-421b6cfa2d28-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:10.281874 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.281594    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:10.281874 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.281602    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6512f069-a146-4131-8f67-421b6cfa2d28-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:10.281874 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.281610    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bng6v\" (UniqueName: \"kubernetes.io/projected/6512f069-a146-4131-8f67-421b6cfa2d28-kube-api-access-bng6v\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:10.936732 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.936699    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66" event={"ID":"6512f069-a146-4131-8f67-421b6cfa2d28","Type":"ContainerDied","Data":"76e4b8a010b6c524a311ff24208d1129aef12c4ce625bc0c922d494b41b22039"}
Sep 17 23:49:10.936732 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.936722    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"
Sep 17 23:49:10.936924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.936785    2571 scope.go:117] "RemoveContainer" containerID="d26b55f14994b690f3741160eca3d889fcae8aa3d93a693b3bdbab641f9bf91c"
Sep 17 23:49:10.945734 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.945716    2571 scope.go:117] "RemoveContainer" containerID="156ec8cdfb79f210fd109a73355bbdf0d52717606fedcf2d5da7a296499598cf"
Sep 17 23:49:10.955058 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.954910    2571 scope.go:117] "RemoveContainer" containerID="b6da171acc3a32b70e65081399da138bedb130b3e838ecb590257e04c2f6de7e"
Sep 17 23:49:10.959581 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.959558    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"]
Sep 17 23:49:10.963727 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:10.963709    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-5gp66"]
Sep 17 23:49:12.057509 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:12.057469    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" path="/var/lib/kubelet/pods/6512f069-a146-4131-8f67-421b6cfa2d28/volumes"
Sep 17 23:49:24.600579 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:24.600546    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:49:27.869149 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:27.869123    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:49:36.094410 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:36.094377    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"]
Sep 17 23:49:36.094994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:36.094589    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kserve-container" containerID="cri-o://ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026" gracePeriod=30
Sep 17 23:49:36.094994 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:36.094621    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kube-rbac-proxy" containerID="cri-o://344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e" gracePeriod=30
Sep 17 23:49:36.118465 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:36.118441    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:49:37.034624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:37.034581    2571 generic.go:356] "Generic (PLEG): container finished" podID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerID="344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e" exitCode=2
Sep 17 23:49:37.034806 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:37.034654    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" event={"ID":"67fe087d-ba6a-4a1f-9cbb-6c3aed120413","Type":"ContainerDied","Data":"344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e"}
Sep 17 23:49:37.860132 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:37.860094    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.47:8643/healthz\": dial tcp 10.133.0.47:8643: connect: connection refused"
Sep 17 23:49:37.866768 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:37.866737    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.47:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.47:8080: connect: connection refused"
Sep 17 23:49:40.633391 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.633367    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:49:40.791455 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.791396    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-cabundle-cert\") pod \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") "
Sep 17 23:49:40.791455 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.791427    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kserve-provision-location\") pod \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") "
Sep 17 23:49:40.791455 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.791450    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxdp4\" (UniqueName: \"kubernetes.io/projected/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-api-access-qxdp4\") pod \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") "
Sep 17 23:49:40.791687 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.791493    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-rbac-proxy-sar-config\") pod \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") "
Sep 17 23:49:40.791687 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.791519    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-proxy-tls\") pod \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\" (UID: \"67fe087d-ba6a-4a1f-9cbb-6c3aed120413\") "
Sep 17 23:49:40.791806 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.791719    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "67fe087d-ba6a-4a1f-9cbb-6c3aed120413" (UID: "67fe087d-ba6a-4a1f-9cbb-6c3aed120413"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:49:40.791894 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.791761    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "67fe087d-ba6a-4a1f-9cbb-6c3aed120413" (UID: "67fe087d-ba6a-4a1f-9cbb-6c3aed120413"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:49:40.791955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.791922    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "67fe087d-ba6a-4a1f-9cbb-6c3aed120413" (UID: "67fe087d-ba6a-4a1f-9cbb-6c3aed120413"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:49:40.793592 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.793567    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "67fe087d-ba6a-4a1f-9cbb-6c3aed120413" (UID: "67fe087d-ba6a-4a1f-9cbb-6c3aed120413"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:49:40.793689 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.793638    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-api-access-qxdp4" (OuterVolumeSpecName: "kube-api-access-qxdp4") pod "67fe087d-ba6a-4a1f-9cbb-6c3aed120413" (UID: "67fe087d-ba6a-4a1f-9cbb-6c3aed120413"). InnerVolumeSpecName "kube-api-access-qxdp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:49:40.892008 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.891984    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:40.892008 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.892011    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:40.892141 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.892021    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:40.892141 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.892029    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:40.892141 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:40.892038    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxdp4\" (UniqueName: \"kubernetes.io/projected/67fe087d-ba6a-4a1f-9cbb-6c3aed120413-kube-api-access-qxdp4\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:49:41.051652 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.051598    2571 generic.go:356] "Generic (PLEG): container finished" podID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerID="ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026" exitCode=0
Sep 17 23:49:41.051743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.051647    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" event={"ID":"67fe087d-ba6a-4a1f-9cbb-6c3aed120413","Type":"ContainerDied","Data":"ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026"}
Sep 17 23:49:41.051743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.051709    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l" event={"ID":"67fe087d-ba6a-4a1f-9cbb-6c3aed120413","Type":"ContainerDied","Data":"f626cc059238d25e1c666ebc7134ad6dd763e979bd86067b5039611bf4f08f88"}
Sep 17 23:49:41.051743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.051656    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"
Sep 17 23:49:41.051743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.051737    2571 scope.go:117] "RemoveContainer" containerID="344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e"
Sep 17 23:49:41.060425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.060408    2571 scope.go:117] "RemoveContainer" containerID="ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026"
Sep 17 23:49:41.067263 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.067243    2571 scope.go:117] "RemoveContainer" containerID="738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46"
Sep 17 23:49:41.074275 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.074261    2571 scope.go:117] "RemoveContainer" containerID="344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e"
Sep 17 23:49:41.074611 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:49:41.074593    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e\": container with ID starting with 344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e not found: ID does not exist" containerID="344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e"
Sep 17 23:49:41.074682 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.074617    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e"} err="failed to get container status \"344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e\": rpc error: code = NotFound desc = could not find container \"344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e\": container with ID starting with 344a7a73fde0cb2c145a0df65fa9db09d8ce8545a3c63429dfe7e237aa94531e not found: ID does not exist"
Sep 17 23:49:41.074682 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.074632    2571 scope.go:117] "RemoveContainer" containerID="ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026"
Sep 17 23:49:41.074859 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.074843    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"]
Sep 17 23:49:41.074859 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:49:41.074853    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026\": container with ID starting with ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026 not found: ID does not exist" containerID="ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026"
Sep 17 23:49:41.074932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.074869    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026"} err="failed to get container status \"ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026\": rpc error: code = NotFound desc = could not find container \"ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026\": container with ID starting with ac0038f9cd2b03bc895a3e65e897c82b8a13dba50a0c06fd19e1f70c2040e026 not found: ID does not exist"
Sep 17 23:49:41.074932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.074882    2571 scope.go:117] "RemoveContainer" containerID="738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46"
Sep 17 23:49:41.075122 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:49:41.075108    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46\": container with ID starting with 738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46 not found: ID does not exist" containerID="738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46"
Sep 17 23:49:41.075166 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.075123    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46"} err="failed to get container status \"738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46\": rpc error: code = NotFound desc = could not find container \"738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46\": container with ID starting with 738b39d5231d8c7f412925cc1187affaa90c8ec7986fd0a3f7c92d9b1198dd46 not found: ID does not exist"
Sep 17 23:49:41.078921 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.078901    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-7t82l"]
Sep 17 23:49:41.263308 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.263283    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:49:41.290698 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.290667    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"]
Sep 17 23:49:41.290993 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.290977    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kube-rbac-proxy"
Sep 17 23:49:41.291070 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.290996    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kube-rbac-proxy"
Sep 17 23:49:41.291070 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291015    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="storage-initializer"
Sep 17 23:49:41.291070 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291024    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="storage-initializer"
Sep 17 23:49:41.291070 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291034    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kserve-container"
Sep 17 23:49:41.291070 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291043    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kserve-container"
Sep 17 23:49:41.291070 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291053    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kube-rbac-proxy"
Sep 17 23:49:41.291070 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291062    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kube-rbac-proxy"
Sep 17 23:49:41.291442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291072    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="storage-initializer"
Sep 17 23:49:41.291442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291080    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="storage-initializer"
Sep 17 23:49:41.291442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291095    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container"
Sep 17 23:49:41.291442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291104    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container"
Sep 17 23:49:41.291442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291178    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kserve-container"
Sep 17 23:49:41.291442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291190    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kube-rbac-proxy"
Sep 17 23:49:41.291442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291199    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6512f069-a146-4131-8f67-421b6cfa2d28" containerName="kserve-container"
Sep 17 23:49:41.291442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.291211    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" containerName="kube-rbac-proxy"
Sep 17 23:49:41.295953 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.295935    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.298489 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.298470    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:49:41.298829 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.298815    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 23:49:41.299000 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.298989    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:49:41.299104 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.299091    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:49:41.299149 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.299126    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:49:41.299232 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.299217    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:49:41.299411 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.299398    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:49:41.303576 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.303559    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"]
Sep 17 23:49:41.396533 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.396510    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.396621 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.396560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.396621 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.396584    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qf98\" (UniqueName: \"kubernetes.io/projected/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-api-access-2qf98\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.396621 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.396609    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.396741 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.396638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.497459 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.497429    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.497597 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.497500    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.497597 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.497540    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.497597 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.497569    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qf98\" (UniqueName: \"kubernetes.io/projected/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-api-access-2qf98\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.497766 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.497606    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.497766 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:49:41.497655    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 17 23:49:41.497766 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:49:41.497720    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls podName:8e5ad78a-8f54-46a6-bad3-e973ea52f35a nodeName:}" failed. No retries permitted until 2026-09-17 23:49:41.997698422 +0000 UTC m=+2306.458616135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-lhckv" (UID: "8e5ad78a-8f54-46a6-bad3-e973ea52f35a") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 17 23:49:41.497926 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.497896    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.498110 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.498090    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.498223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.498204    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.511444 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.511423    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qf98\" (UniqueName: \"kubernetes.io/projected/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-api-access-2qf98\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:41.645845 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:41.645819    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:49:42.001433 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:42.001366    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:42.003710 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:42.003682    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-lhckv\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:42.057518 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:42.057487    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67fe087d-ba6a-4a1f-9cbb-6c3aed120413" path="/var/lib/kubelet/pods/67fe087d-ba6a-4a1f-9cbb-6c3aed120413/volumes"
Sep 17 23:49:42.206642 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:42.206624    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:49:42.478224 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:42.478183    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"]
Sep 17 23:49:42.480913 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:49:42.480889    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5ad78a_8f54_46a6_bad3_e973ea52f35a.slice/crio-9965f271c36d4f9fbb45c237c010f53e16d5e6ef93fe35fa552c8d7609f42a86 WatchSource:0}: Error finding container 9965f271c36d4f9fbb45c237c010f53e16d5e6ef93fe35fa552c8d7609f42a86: Status 404 returned error can't find the container with id 9965f271c36d4f9fbb45c237c010f53e16d5e6ef93fe35fa552c8d7609f42a86
Sep 17 23:49:43.061397 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:43.061361    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" event={"ID":"8e5ad78a-8f54-46a6-bad3-e973ea52f35a","Type":"ContainerStarted","Data":"aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d"}
Sep 17 23:49:43.061730 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:43.061430    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" event={"ID":"8e5ad78a-8f54-46a6-bad3-e973ea52f35a","Type":"ContainerStarted","Data":"9965f271c36d4f9fbb45c237c010f53e16d5e6ef93fe35fa552c8d7609f42a86"}
Sep 17 23:49:47.077380 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:47.077319    2571 generic.go:356] "Generic (PLEG): container finished" podID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerID="aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d" exitCode=0
Sep 17 23:49:47.077826 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:49:47.077397    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" event={"ID":"8e5ad78a-8f54-46a6-bad3-e973ea52f35a","Type":"ContainerDied","Data":"aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d"}
Sep 17 23:50:04.158093 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:04.158061    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" event={"ID":"8e5ad78a-8f54-46a6-bad3-e973ea52f35a","Type":"ContainerStarted","Data":"360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96"}
Sep 17 23:50:04.158497 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:04.158126    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" event={"ID":"8e5ad78a-8f54-46a6-bad3-e973ea52f35a","Type":"ContainerStarted","Data":"29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd"}
Sep 17 23:50:04.158497 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:04.158348    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:50:04.177668 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:04.177631    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podStartSLOduration=2.480826062 podStartE2EDuration="23.177617182s" totalImagesPullingTime="16.851905865s" totalInitContainerRuntime="3.844885255s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:50:04.176667238 +0000 UTC m=+2328.637584968" watchObservedRunningTime="2026-09-17 23:50:04.177617182 +0000 UTC m=+2328.638534913"
Sep 17 23:50:05.167685 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:05.167641    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:50:05.169056 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:05.169027    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 23:50:06.171407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:06.171364    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 23:50:11.176408 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:11.176339    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:50:11.176842 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:11.176819    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 23:50:21.176961 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:21.176924    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 23:50:31.177448 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:31.177405    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 23:50:41.177486 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:41.177450    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 23:50:51.176867 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:50:51.176829    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 23:51:01.109612 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:01.109581    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:01.177118 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:01.177082    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Sep 17 23:51:06.253408 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.253377    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:06.277357 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.277311    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"]
Sep 17 23:51:06.280820 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.280805    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.283257 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.283232    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:51:06.283397 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.283314    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 23:51:06.290706 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.290686    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"]
Sep 17 23:51:06.397591 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.397569    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjjqf\" (UniqueName: \"kubernetes.io/projected/93392933-bb39-4273-8e56-4f55979ae191-kube-api-access-qjjqf\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.397692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.397595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93392933-bb39-4273-8e56-4f55979ae191-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.397692 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.397638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.397783 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.397705    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.397783 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.397743    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.498896 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.498877    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.498993 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.498908    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.498993 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.498934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.498993 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.498977    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjjqf\" (UniqueName: \"kubernetes.io/projected/93392933-bb39-4273-8e56-4f55979ae191-kube-api-access-qjjqf\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.499131 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.499002    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93392933-bb39-4273-8e56-4f55979ae191-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.499131 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:51:06.499058    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 17 23:51:06.499131 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:51:06.499127    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls podName:93392933-bb39-4273-8e56-4f55979ae191 nodeName:}" failed. No retries permitted until 2026-09-17 23:51:06.999107926 +0000 UTC m=+2391.460025635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" (UID: "93392933-bb39-4273-8e56-4f55979ae191") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 17 23:51:06.499364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.499346    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93392933-bb39-4273-8e56-4f55979ae191-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.499578 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.499563    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.499620 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.499587    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.507861 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.507812    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjjqf\" (UniqueName: \"kubernetes.io/projected/93392933-bb39-4273-8e56-4f55979ae191-kube-api-access-qjjqf\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:06.900421 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:06.900399    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:07.003341 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:07.003308    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:07.005533 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:07.005517    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:07.191182 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:07.191135    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:07.461932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:07.461905    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"]
Sep 17 23:51:07.464391 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:51:07.464364    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93392933_bb39_4273_8e56_4f55979ae191.slice/crio-2afa5901961284f03ef262b96e980d2d586fbeaf718cba114516926ab3c95649 WatchSource:0}: Error finding container 2afa5901961284f03ef262b96e980d2d586fbeaf718cba114516926ab3c95649: Status 404 returned error can't find the container with id 2afa5901961284f03ef262b96e980d2d586fbeaf718cba114516926ab3c95649
Sep 17 23:51:08.402281 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:08.402246    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" event={"ID":"93392933-bb39-4273-8e56-4f55979ae191","Type":"ContainerStarted","Data":"5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df"}
Sep 17 23:51:08.402484 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:08.402316    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" event={"ID":"93392933-bb39-4273-8e56-4f55979ae191","Type":"ContainerStarted","Data":"2afa5901961284f03ef262b96e980d2d586fbeaf718cba114516926ab3c95649"}
Sep 17 23:51:11.177458 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:11.177429    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:51:11.415564 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:11.415533    2571 generic.go:356] "Generic (PLEG): container finished" podID="93392933-bb39-4273-8e56-4f55979ae191" containerID="5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df" exitCode=0
Sep 17 23:51:11.415683 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:11.415583    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" event={"ID":"93392933-bb39-4273-8e56-4f55979ae191","Type":"ContainerDied","Data":"5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df"}
Sep 17 23:51:12.422448 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:12.422415    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" event={"ID":"93392933-bb39-4273-8e56-4f55979ae191","Type":"ContainerStarted","Data":"05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4"}
Sep 17 23:51:12.422809 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:12.422474    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" event={"ID":"93392933-bb39-4273-8e56-4f55979ae191","Type":"ContainerStarted","Data":"9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6"}
Sep 17 23:51:12.422809 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:12.422682    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:12.422809 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:12.422728    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:12.442768 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:12.442723    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" podStartSLOduration=2.621281784 podStartE2EDuration="6.442711128s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.821429344s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:51:12.441342158 +0000 UTC m=+2396.902259881" watchObservedRunningTime="2026-09-17 23:51:12.442711128 +0000 UTC m=+2396.903628858"
Sep 17 23:51:18.431511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:18.431475    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:21.841844 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:21.841814    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"]
Sep 17 23:51:21.842243 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:21.842037    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container" containerID="cri-o://29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd" gracePeriod=30
Sep 17 23:51:21.842243 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:21.842066    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kube-rbac-proxy" containerID="cri-o://360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96" gracePeriod=30
Sep 17 23:51:21.873106 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:21.873073    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:22.466264 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:22.466231    2571 generic.go:356] "Generic (PLEG): container finished" podID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerID="360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96" exitCode=2
Sep 17 23:51:22.466497 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:22.466311    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" event={"ID":"8e5ad78a-8f54-46a6-bad3-e973ea52f35a","Type":"ContainerDied","Data":"360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96"}
Sep 17 23:51:24.875620 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.875596    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:51:24.932612 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.932587    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-cabundle-cert\") pod \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") "
Sep 17 23:51:24.932725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.932632    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qf98\" (UniqueName: \"kubernetes.io/projected/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-api-access-2qf98\") pod \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") "
Sep 17 23:51:24.932725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.932658    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls\") pod \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") "
Sep 17 23:51:24.932725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.932697    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kserve-provision-location\") pod \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") "
Sep 17 23:51:24.932855 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.932721    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-rbac-proxy-sar-config\") pod \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\" (UID: \"8e5ad78a-8f54-46a6-bad3-e973ea52f35a\") "
Sep 17 23:51:24.932974 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.932950    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e5ad78a-8f54-46a6-bad3-e973ea52f35a" (UID: "8e5ad78a-8f54-46a6-bad3-e973ea52f35a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:51:24.933053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.933026    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e5ad78a-8f54-46a6-bad3-e973ea52f35a" (UID: "8e5ad78a-8f54-46a6-bad3-e973ea52f35a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:51:24.933121 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.933101    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e5ad78a-8f54-46a6-bad3-e973ea52f35a" (UID: "8e5ad78a-8f54-46a6-bad3-e973ea52f35a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:51:24.934733 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.934710    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e5ad78a-8f54-46a6-bad3-e973ea52f35a" (UID: "8e5ad78a-8f54-46a6-bad3-e973ea52f35a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:51:24.934829 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:24.934808    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-api-access-2qf98" (OuterVolumeSpecName: "kube-api-access-2qf98") pod "8e5ad78a-8f54-46a6-bad3-e973ea52f35a" (UID: "8e5ad78a-8f54-46a6-bad3-e973ea52f35a"). InnerVolumeSpecName "kube-api-access-2qf98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:51:25.033265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.033217    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:25.033265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.033236    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qf98\" (UniqueName: \"kubernetes.io/projected/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-api-access-2qf98\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:25.033265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.033245    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:25.033265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.033254    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:25.033265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.033262    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e5ad78a-8f54-46a6-bad3-e973ea52f35a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:25.480513 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.480482    2571 generic.go:356] "Generic (PLEG): container finished" podID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerID="29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd" exitCode=0
Sep 17 23:51:25.480632 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.480529    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"
Sep 17 23:51:25.480632 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.480570    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" event={"ID":"8e5ad78a-8f54-46a6-bad3-e973ea52f35a","Type":"ContainerDied","Data":"29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd"}
Sep 17 23:51:25.480632 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.480625    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv" event={"ID":"8e5ad78a-8f54-46a6-bad3-e973ea52f35a","Type":"ContainerDied","Data":"9965f271c36d4f9fbb45c237c010f53e16d5e6ef93fe35fa552c8d7609f42a86"}
Sep 17 23:51:25.480739 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.480652    2571 scope.go:117] "RemoveContainer" containerID="360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96"
Sep 17 23:51:25.489364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.489346    2571 scope.go:117] "RemoveContainer" containerID="29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd"
Sep 17 23:51:25.496514 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.496494    2571 scope.go:117] "RemoveContainer" containerID="aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d"
Sep 17 23:51:25.502162 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.502137    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"]
Sep 17 23:51:25.503905 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.503892    2571 scope.go:117] "RemoveContainer" containerID="360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96"
Sep 17 23:51:25.504140 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:51:25.504125    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96\": container with ID starting with 360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96 not found: ID does not exist" containerID="360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96"
Sep 17 23:51:25.504185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.504147    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96"} err="failed to get container status \"360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96\": rpc error: code = NotFound desc = could not find container \"360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96\": container with ID starting with 360a105cba4d8e652f3106115bf6337205ba1210c17e41b987d56db981436e96 not found: ID does not exist"
Sep 17 23:51:25.504185 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.504162    2571 scope.go:117] "RemoveContainer" containerID="29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd"
Sep 17 23:51:25.504386 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:51:25.504369    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd\": container with ID starting with 29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd not found: ID does not exist" containerID="29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd"
Sep 17 23:51:25.504434 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.504391    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd"} err="failed to get container status \"29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd\": rpc error: code = NotFound desc = could not find container \"29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd\": container with ID starting with 29f06f7a660eac6955047f687179f84e92c5bdf0dbf516c92b852a8f7bf1f9fd not found: ID does not exist"
Sep 17 23:51:25.504434 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.504403    2571 scope.go:117] "RemoveContainer" containerID="aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d"
Sep 17 23:51:25.504647 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:51:25.504633    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d\": container with ID starting with aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d not found: ID does not exist" containerID="aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d"
Sep 17 23:51:25.504702 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.504659    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d"} err="failed to get container status \"aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d\": rpc error: code = NotFound desc = could not find container \"aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d\": container with ID starting with aa5b99d354e0fae9d08bf4125968053534a23a31510311bd009089914fbe810d not found: ID does not exist"
Sep 17 23:51:25.507214 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:25.507190    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-lhckv"]
Sep 17 23:51:26.057346 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:26.057287    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" path="/var/lib/kubelet/pods/8e5ad78a-8f54-46a6-bad3-e973ea52f35a/volumes"
Sep 17 23:51:26.999918 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:26.999879    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:27.029687 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.029665    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"]
Sep 17 23:51:27.029951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.029940    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="storage-initializer"
Sep 17 23:51:27.029999 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.029953    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="storage-initializer"
Sep 17 23:51:27.029999 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.029966    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kube-rbac-proxy"
Sep 17 23:51:27.029999 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.029971    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kube-rbac-proxy"
Sep 17 23:51:27.029999 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.029983    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container"
Sep 17 23:51:27.029999 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.029988    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container"
Sep 17 23:51:27.030169 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.030039    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kserve-container"
Sep 17 23:51:27.030169 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.030049    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5ad78a-8f54-46a6-bad3-e973ea52f35a" containerName="kube-rbac-proxy"
Sep 17 23:51:27.034306 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.034291    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.037051 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.037029    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 23:51:27.037051 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.037046    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:51:27.043904 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.043884    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"]
Sep 17 23:51:27.047474 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.047450    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pq72\" (UniqueName: \"kubernetes.io/projected/689cdabc-efeb-4a04-b033-c53212f91c31-kube-api-access-7pq72\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.047572 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.047489    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.047648 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.047629    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689cdabc-efeb-4a04-b033-c53212f91c31-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.047709 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.047673    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689cdabc-efeb-4a04-b033-c53212f91c31-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.047763 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.047726    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.106940 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.106900    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:27.148110 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.148092    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pq72\" (UniqueName: \"kubernetes.io/projected/689cdabc-efeb-4a04-b033-c53212f91c31-kube-api-access-7pq72\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.148197 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.148118    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.148197 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.148160    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689cdabc-efeb-4a04-b033-c53212f91c31-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.148197 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.148191    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689cdabc-efeb-4a04-b033-c53212f91c31-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.148387 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.148219    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.148645 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.148621    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689cdabc-efeb-4a04-b033-c53212f91c31-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.148903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.148883    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.148947 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.148884    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.150642 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.150627    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689cdabc-efeb-4a04-b033-c53212f91c31-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.157248 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.157231    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pq72\" (UniqueName: \"kubernetes.io/projected/689cdabc-efeb-4a04-b033-c53212f91c31-kube-api-access-7pq72\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.344015 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.343998    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:27.614588 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:27.614563    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"]
Sep 17 23:51:27.615526 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:51:27.615498    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod689cdabc_efeb_4a04_b033_c53212f91c31.slice/crio-df2dead8b100e2f75e4383d6d01676bae64aab6a1cc608317ca987c37e3f5cfb WatchSource:0}: Error finding container df2dead8b100e2f75e4383d6d01676bae64aab6a1cc608317ca987c37e3f5cfb: Status 404 returned error can't find the container with id df2dead8b100e2f75e4383d6d01676bae64aab6a1cc608317ca987c37e3f5cfb
Sep 17 23:51:28.493755 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:28.493723    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" event={"ID":"689cdabc-efeb-4a04-b033-c53212f91c31","Type":"ContainerStarted","Data":"9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05"}
Sep 17 23:51:28.494113 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:28.493777    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" event={"ID":"689cdabc-efeb-4a04-b033-c53212f91c31","Type":"ContainerStarted","Data":"df2dead8b100e2f75e4383d6d01676bae64aab6a1cc608317ca987c37e3f5cfb"}
Sep 17 23:51:32.512837 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:32.512807    2571 generic.go:356] "Generic (PLEG): container finished" podID="689cdabc-efeb-4a04-b033-c53212f91c31" containerID="9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05" exitCode=0
Sep 17 23:51:32.513215 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:32.512866    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" event={"ID":"689cdabc-efeb-4a04-b033-c53212f91c31","Type":"ContainerDied","Data":"9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05"}
Sep 17 23:51:33.519024 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:33.518992    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" event={"ID":"689cdabc-efeb-4a04-b033-c53212f91c31","Type":"ContainerStarted","Data":"cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b"}
Sep 17 23:51:33.519414 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:33.519052    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" event={"ID":"689cdabc-efeb-4a04-b033-c53212f91c31","Type":"ContainerStarted","Data":"81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23"}
Sep 17 23:51:33.519414 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:33.519244    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:33.539930 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:33.539890    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" podStartSLOduration=2.68821783 podStartE2EDuration="6.53987936s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.85166153s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:51:33.53875004 +0000 UTC m=+2417.999667775" watchObservedRunningTime="2026-09-17 23:51:33.53987936 +0000 UTC m=+2418.000797089"
Sep 17 23:51:34.523162 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:34.523128    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:37.982284 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:37.982185    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:51:38.004623 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:37.985783    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:51:40.532123 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:40.532096    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:51:45.902510 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:45.902477    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:45.982236 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:45.982213    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:48.434904 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:48.434877    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:56.355381 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:56.355345    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"]
Sep 17 23:51:56.355869 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:56.355554    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="kserve-container" containerID="cri-o://9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6" gracePeriod=30
Sep 17 23:51:56.355869 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:56.355587    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="kube-rbac-proxy" containerID="cri-o://05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4" gracePeriod=30
Sep 17 23:51:56.386913 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:56.386889    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:51:56.610860 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:56.610778    2571 generic.go:356] "Generic (PLEG): container finished" podID="93392933-bb39-4273-8e56-4f55979ae191" containerID="05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4" exitCode=2
Sep 17 23:51:56.610860 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:56.610853    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" event={"ID":"93392933-bb39-4273-8e56-4f55979ae191","Type":"ContainerDied","Data":"05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4"}
Sep 17 23:51:57.291110 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.291089    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:57.359450 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.359392    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls\") pod \"93392933-bb39-4273-8e56-4f55979ae191\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") "
Sep 17 23:51:57.359450 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.359419    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-cabundle-cert\") pod \"93392933-bb39-4273-8e56-4f55979ae191\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") "
Sep 17 23:51:57.359450 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.359442    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93392933-bb39-4273-8e56-4f55979ae191-kserve-provision-location\") pod \"93392933-bb39-4273-8e56-4f55979ae191\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") "
Sep 17 23:51:57.359897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.359477    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjjqf\" (UniqueName: \"kubernetes.io/projected/93392933-bb39-4273-8e56-4f55979ae191-kube-api-access-qjjqf\") pod \"93392933-bb39-4273-8e56-4f55979ae191\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") "
Sep 17 23:51:57.359897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.359513    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-kube-rbac-proxy-sar-config\") pod \"93392933-bb39-4273-8e56-4f55979ae191\" (UID: \"93392933-bb39-4273-8e56-4f55979ae191\") "
Sep 17 23:51:57.359897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.359728    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93392933-bb39-4273-8e56-4f55979ae191-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93392933-bb39-4273-8e56-4f55979ae191" (UID: "93392933-bb39-4273-8e56-4f55979ae191"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:51:57.359897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.359768    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93392933-bb39-4273-8e56-4f55979ae191" (UID: "93392933-bb39-4273-8e56-4f55979ae191"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:51:57.360041 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.359916    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "93392933-bb39-4273-8e56-4f55979ae191" (UID: "93392933-bb39-4273-8e56-4f55979ae191"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:51:57.361596 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.361575    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93392933-bb39-4273-8e56-4f55979ae191-kube-api-access-qjjqf" (OuterVolumeSpecName: "kube-api-access-qjjqf") pod "93392933-bb39-4273-8e56-4f55979ae191" (UID: "93392933-bb39-4273-8e56-4f55979ae191"). InnerVolumeSpecName "kube-api-access-qjjqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:51:57.361658 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.361602    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93392933-bb39-4273-8e56-4f55979ae191" (UID: "93392933-bb39-4273-8e56-4f55979ae191"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:51:57.460511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.460489    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjjqf\" (UniqueName: \"kubernetes.io/projected/93392933-bb39-4273-8e56-4f55979ae191-kube-api-access-qjjqf\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:57.460511 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.460508    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:57.460626 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.460518    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93392933-bb39-4273-8e56-4f55979ae191-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:57.460626 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.460526    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93392933-bb39-4273-8e56-4f55979ae191-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:57.460626 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.460533    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93392933-bb39-4273-8e56-4f55979ae191-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:51:57.616725 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.616668    2571 generic.go:356] "Generic (PLEG): container finished" podID="93392933-bb39-4273-8e56-4f55979ae191" containerID="9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6" exitCode=0
Sep 17 23:51:57.616802 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.616729    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"
Sep 17 23:51:57.616802 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.616735    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" event={"ID":"93392933-bb39-4273-8e56-4f55979ae191","Type":"ContainerDied","Data":"9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6"}
Sep 17 23:51:57.616802 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.616795    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz" event={"ID":"93392933-bb39-4273-8e56-4f55979ae191","Type":"ContainerDied","Data":"2afa5901961284f03ef262b96e980d2d586fbeaf718cba114516926ab3c95649"}
Sep 17 23:51:57.616912 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.616821    2571 scope.go:117] "RemoveContainer" containerID="05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4"
Sep 17 23:51:57.628135 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.628106    2571 scope.go:117] "RemoveContainer" containerID="9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6"
Sep 17 23:51:57.636948 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.636933    2571 scope.go:117] "RemoveContainer" containerID="5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df"
Sep 17 23:51:57.644041 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.643979    2571 scope.go:117] "RemoveContainer" containerID="05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4"
Sep 17 23:51:57.644295 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:51:57.644277    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4\": container with ID starting with 05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4 not found: ID does not exist" containerID="05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4"
Sep 17 23:51:57.644417 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.644304    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4"} err="failed to get container status \"05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4\": rpc error: code = NotFound desc = could not find container \"05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4\": container with ID starting with 05cb8e7c8ec823ef24ab6670b0fea04b22af3a30f453c7cd8286b60687e41ab4 not found: ID does not exist"
Sep 17 23:51:57.644476 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.644425    2571 scope.go:117] "RemoveContainer" containerID="9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6"
Sep 17 23:51:57.644710 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:51:57.644693    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6\": container with ID starting with 9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6 not found: ID does not exist" containerID="9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6"
Sep 17 23:51:57.644772 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.644715    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6"} err="failed to get container status \"9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6\": rpc error: code = NotFound desc = could not find container \"9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6\": container with ID starting with 9f970d2ee0efdc00957a438f68bb5e94c42f46f4e37398d5f66b598a5769d4f6 not found: ID does not exist"
Sep 17 23:51:57.644772 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.644731    2571 scope.go:117] "RemoveContainer" containerID="5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df"
Sep 17 23:51:57.644982 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:51:57.644968    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df\": container with ID starting with 5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df not found: ID does not exist" containerID="5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df"
Sep 17 23:51:57.645029 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.644986    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df"} err="failed to get container status \"5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df\": rpc error: code = NotFound desc = could not find container \"5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df\": container with ID starting with 5313eadbcd97618e3d4856321a9b4b7f419fac07b740f70d201ed4551c17b8df not found: ID does not exist"
Sep 17 23:51:57.645029 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.645009    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"]
Sep 17 23:51:57.648883 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:57.648864    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-j24rz"]
Sep 17 23:51:58.058096 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:51:58.058027    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93392933-bb39-4273-8e56-4f55979ae191" path="/var/lib/kubelet/pods/93392933-bb39-4273-8e56-4f55979ae191/volumes"
Sep 17 23:52:01.522603 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:01.522574    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:52:01.794846 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:01.794760    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:52:10.535767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:10.535737    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:52:17.135821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:17.135791    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"]
Sep 17 23:52:17.136188 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:17.136038    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kube-rbac-proxy" containerID="cri-o://cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b" gracePeriod=30
Sep 17 23:52:17.136188 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:17.136070    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kserve-container" containerID="cri-o://81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23" gracePeriod=30
Sep 17 23:52:17.180942 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:17.180918    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:52:17.692092 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:17.692054    2571 generic.go:356] "Generic (PLEG): container finished" podID="689cdabc-efeb-4a04-b033-c53212f91c31" containerID="cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b" exitCode=2
Sep 17 23:52:17.692245 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:17.692126    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" event={"ID":"689cdabc-efeb-4a04-b033-c53212f91c31","Type":"ContainerDied","Data":"cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b"}
Sep 17 23:52:20.527800 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:20.527759    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.50:8643/healthz\": dial tcp 10.133.0.50:8643: connect: connection refused"
Sep 17 23:52:20.533336 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:20.533295    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.50:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.133.0.50:8080: connect: connection refused"
Sep 17 23:52:22.265549 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.265519    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:52:22.474910 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.474887    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:52:22.542947 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.542888    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:52:22.635259 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.635240    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pq72\" (UniqueName: \"kubernetes.io/projected/689cdabc-efeb-4a04-b033-c53212f91c31-kube-api-access-7pq72\") pod \"689cdabc-efeb-4a04-b033-c53212f91c31\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") "
Sep 17 23:52:22.635379 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.635278    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689cdabc-efeb-4a04-b033-c53212f91c31-kserve-provision-location\") pod \"689cdabc-efeb-4a04-b033-c53212f91c31\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") "
Sep 17 23:52:22.635379 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.635303    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-kube-rbac-proxy-sar-config\") pod \"689cdabc-efeb-4a04-b033-c53212f91c31\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") "
Sep 17 23:52:22.635379 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.635349    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689cdabc-efeb-4a04-b033-c53212f91c31-proxy-tls\") pod \"689cdabc-efeb-4a04-b033-c53212f91c31\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") "
Sep 17 23:52:22.635379 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.635378    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-cabundle-cert\") pod \"689cdabc-efeb-4a04-b033-c53212f91c31\" (UID: \"689cdabc-efeb-4a04-b033-c53212f91c31\") "
Sep 17 23:52:22.635623 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.635600    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/689cdabc-efeb-4a04-b033-c53212f91c31-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "689cdabc-efeb-4a04-b033-c53212f91c31" (UID: "689cdabc-efeb-4a04-b033-c53212f91c31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:52:22.635755 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.635734    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "689cdabc-efeb-4a04-b033-c53212f91c31" (UID: "689cdabc-efeb-4a04-b033-c53212f91c31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:52:22.635814 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.635759    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "689cdabc-efeb-4a04-b033-c53212f91c31" (UID: "689cdabc-efeb-4a04-b033-c53212f91c31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:52:22.637403 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.637373    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/689cdabc-efeb-4a04-b033-c53212f91c31-kube-api-access-7pq72" (OuterVolumeSpecName: "kube-api-access-7pq72") pod "689cdabc-efeb-4a04-b033-c53212f91c31" (UID: "689cdabc-efeb-4a04-b033-c53212f91c31"). InnerVolumeSpecName "kube-api-access-7pq72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:52:22.637518 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.637439    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/689cdabc-efeb-4a04-b033-c53212f91c31-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "689cdabc-efeb-4a04-b033-c53212f91c31" (UID: "689cdabc-efeb-4a04-b033-c53212f91c31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:52:22.713223 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.713201    2571 generic.go:356] "Generic (PLEG): container finished" podID="689cdabc-efeb-4a04-b033-c53212f91c31" containerID="81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23" exitCode=0
Sep 17 23:52:22.713316 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.713252    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"
Sep 17 23:52:22.713316 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.713262    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" event={"ID":"689cdabc-efeb-4a04-b033-c53212f91c31","Type":"ContainerDied","Data":"81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23"}
Sep 17 23:52:22.713413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.713313    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv" event={"ID":"689cdabc-efeb-4a04-b033-c53212f91c31","Type":"ContainerDied","Data":"df2dead8b100e2f75e4383d6d01676bae64aab6a1cc608317ca987c37e3f5cfb"}
Sep 17 23:52:22.713413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.713358    2571 scope.go:117] "RemoveContainer" containerID="cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b"
Sep 17 23:52:22.721645 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.721633    2571 scope.go:117] "RemoveContainer" containerID="81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23"
Sep 17 23:52:22.728715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.728699    2571 scope.go:117] "RemoveContainer" containerID="9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05"
Sep 17 23:52:22.737086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.736985    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"]
Sep 17 23:52:22.740073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.739080    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pq72\" (UniqueName: \"kubernetes.io/projected/689cdabc-efeb-4a04-b033-c53212f91c31-kube-api-access-7pq72\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:52:22.740073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.739106    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/689cdabc-efeb-4a04-b033-c53212f91c31-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:52:22.740073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.739128    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:52:22.740073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.739143    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/689cdabc-efeb-4a04-b033-c53212f91c31-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:52:22.740073 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.739157    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/689cdabc-efeb-4a04-b033-c53212f91c31-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:52:22.741432 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.741411    2571 scope.go:117] "RemoveContainer" containerID="cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b"
Sep 17 23:52:22.741711 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:52:22.741681    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b\": container with ID starting with cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b not found: ID does not exist" containerID="cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b"
Sep 17 23:52:22.741799 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.741721    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b"} err="failed to get container status \"cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b\": rpc error: code = NotFound desc = could not find container \"cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b\": container with ID starting with cd1299cd70e06e1292f4976ca76912ebcc28d06a4c51831c67576bc39290dd9b not found: ID does not exist"
Sep 17 23:52:22.741799 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.741747    2571 scope.go:117] "RemoveContainer" containerID="81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23"
Sep 17 23:52:22.741985 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.741970    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-b2wfv"]
Sep 17 23:52:22.742025 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:52:22.742010    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23\": container with ID starting with 81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23 not found: ID does not exist" containerID="81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23"
Sep 17 23:52:22.742103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.742030    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23"} err="failed to get container status \"81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23\": rpc error: code = NotFound desc = could not find container \"81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23\": container with ID starting with 81bdb2c1f820351bb23903c1020f861156cec74b95fb375d60af15cc20e21a23 not found: ID does not exist"
Sep 17 23:52:22.742103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.742046    2571 scope.go:117] "RemoveContainer" containerID="9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05"
Sep 17 23:52:22.742315 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:52:22.742296    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05\": container with ID starting with 9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05 not found: ID does not exist" containerID="9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05"
Sep 17 23:52:22.742652 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:22.742344    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05"} err="failed to get container status \"9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05\": rpc error: code = NotFound desc = could not find container \"9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05\": container with ID starting with 9b666f14eb925cfd45a9aee7d5c6410c06caba61d70d46ced624a76b24d0ac05 not found: ID does not exist"
Sep 17 23:52:24.057724 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:24.057698    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" path="/var/lib/kubelet/pods/689cdabc-efeb-4a04-b033-c53212f91c31/volumes"
Sep 17 23:52:31.676440 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:31.676405    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:52:36.829634 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.829598    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:52:36.838903 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.838869    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"]
Sep 17 23:52:36.839507 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839485    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="storage-initializer"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839510    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="storage-initializer"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839531    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="kube-rbac-proxy"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839540    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="kube-rbac-proxy"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839550    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kube-rbac-proxy"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839558    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kube-rbac-proxy"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839576    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="kserve-container"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839584    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="kserve-container"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839612    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="storage-initializer"
Sep 17 23:52:36.839625 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839621    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="storage-initializer"
Sep 17 23:52:36.840052 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839635    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kserve-container"
Sep 17 23:52:36.840052 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839643    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kserve-container"
Sep 17 23:52:36.840052 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839762    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kube-rbac-proxy"
Sep 17 23:52:36.840052 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839779    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="kube-rbac-proxy"
Sep 17 23:52:36.840052 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839792    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="689cdabc-efeb-4a04-b033-c53212f91c31" containerName="kserve-container"
Sep 17 23:52:36.840052 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.839805    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="93392933-bb39-4273-8e56-4f55979ae191" containerName="kserve-container"
Sep 17 23:52:36.844245 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.844227    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:36.847186 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.847166    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:52:36.847314 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.847166    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:52:36.847314 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.847267    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 23:52:36.847314 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.847166    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:52:36.847314 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.847171    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 23:52:36.848612 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.848541    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:52:36.848612 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.848605    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:52:36.850766 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.850746    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"]
Sep 17 23:52:36.934838 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.934816    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8rmx\" (UniqueName: \"kubernetes.io/projected/26617538-bc30-43ce-86d6-5bb08ad00551-kube-api-access-m8rmx\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:36.934958 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.934846    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:36.934958 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.934863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:36.934958 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.934878    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26617538-bc30-43ce-86d6-5bb08ad00551-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:36.935064 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.934971    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26617538-bc30-43ce-86d6-5bb08ad00551-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:36.956339 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:36.956307    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:52:37.036161 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.036140    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26617538-bc30-43ce-86d6-5bb08ad00551-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.036228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.036182    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8rmx\" (UniqueName: \"kubernetes.io/projected/26617538-bc30-43ce-86d6-5bb08ad00551-kube-api-access-m8rmx\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.036228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.036200    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.036228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.036223    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.036370 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.036348    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26617538-bc30-43ce-86d6-5bb08ad00551-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.036677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.036653    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26617538-bc30-43ce-86d6-5bb08ad00551-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.036868 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.036851    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.036932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.036865    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.038669 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.038653    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26617538-bc30-43ce-86d6-5bb08ad00551-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.045892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.045875    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8rmx\" (UniqueName: \"kubernetes.io/projected/26617538-bc30-43ce-86d6-5bb08ad00551-kube-api-access-m8rmx\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.155362 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.155342    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:37.426836 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.426812    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"]
Sep 17 23:52:37.429318 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:52:37.429294    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26617538_bc30_43ce_86d6_5bb08ad00551.slice/crio-0dc0408106fa78b0269436789349e4836655c29543ed783d964c387770f0f142 WatchSource:0}: Error finding container 0dc0408106fa78b0269436789349e4836655c29543ed783d964c387770f0f142: Status 404 returned error can't find the container with id 0dc0408106fa78b0269436789349e4836655c29543ed783d964c387770f0f142
Sep 17 23:52:37.770619 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.770530    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" event={"ID":"26617538-bc30-43ce-86d6-5bb08ad00551","Type":"ContainerStarted","Data":"a9a9a31d7b7d86c667ee8705082fa2bf756509c55b05f3abf983a7e6694e00a6"}
Sep 17 23:52:37.770619 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:37.770607    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" event={"ID":"26617538-bc30-43ce-86d6-5bb08ad00551","Type":"ContainerStarted","Data":"0dc0408106fa78b0269436789349e4836655c29543ed783d964c387770f0f142"}
Sep 17 23:52:41.786682 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:41.786654    2571 generic.go:356] "Generic (PLEG): container finished" podID="26617538-bc30-43ce-86d6-5bb08ad00551" containerID="a9a9a31d7b7d86c667ee8705082fa2bf756509c55b05f3abf983a7e6694e00a6" exitCode=0
Sep 17 23:52:41.786995 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:41.786723    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" event={"ID":"26617538-bc30-43ce-86d6-5bb08ad00551","Type":"ContainerDied","Data":"a9a9a31d7b7d86c667ee8705082fa2bf756509c55b05f3abf983a7e6694e00a6"}
Sep 17 23:52:42.793483 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:42.793445    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" event={"ID":"26617538-bc30-43ce-86d6-5bb08ad00551","Type":"ContainerStarted","Data":"13671bbc0832701707b16e899c4ba0e9e51a6cd12409eeaad90fbe933be55636"}
Sep 17 23:52:42.793835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:42.793511    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" event={"ID":"26617538-bc30-43ce-86d6-5bb08ad00551","Type":"ContainerStarted","Data":"1948222b58f2515b0eadb3f2c8c93b0930b191fa7717a3321166eb90e239f216"}
Sep 17 23:52:42.793835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:42.793737    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:42.793916 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:42.793898    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:42.815089 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:42.815046    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" podStartSLOduration=2.564268986 podStartE2EDuration="6.81503436s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.250765374s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:52:42.813428775 +0000 UTC m=+2487.274346506" watchObservedRunningTime="2026-09-17 23:52:42.81503436 +0000 UTC m=+2487.275952092"
Sep 17 23:52:48.803111 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:48.803081    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:52:58.956290 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:52:58.956250    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:18.806915 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:18.806884    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:53:26.917644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:26.917604    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"]
Sep 17 23:53:26.918132 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:26.917907    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="kserve-container" containerID="cri-o://1948222b58f2515b0eadb3f2c8c93b0930b191fa7717a3321166eb90e239f216" gracePeriod=30
Sep 17 23:53:26.918132 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:26.917977    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="kube-rbac-proxy" containerID="cri-o://13671bbc0832701707b16e899c4ba0e9e51a6cd12409eeaad90fbe933be55636" gracePeriod=30
Sep 17 23:53:26.944183 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:26.944162    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:27.965704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:27.965679    2571 generic.go:356] "Generic (PLEG): container finished" podID="26617538-bc30-43ce-86d6-5bb08ad00551" containerID="13671bbc0832701707b16e899c4ba0e9e51a6cd12409eeaad90fbe933be55636" exitCode=2
Sep 17 23:53:27.965704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:27.965704    2571 generic.go:356] "Generic (PLEG): container finished" podID="26617538-bc30-43ce-86d6-5bb08ad00551" containerID="1948222b58f2515b0eadb3f2c8c93b0930b191fa7717a3321166eb90e239f216" exitCode=0
Sep 17 23:53:27.966036 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:27.965743    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" event={"ID":"26617538-bc30-43ce-86d6-5bb08ad00551","Type":"ContainerDied","Data":"13671bbc0832701707b16e899c4ba0e9e51a6cd12409eeaad90fbe933be55636"}
Sep 17 23:53:27.966036 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:27.965821    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" event={"ID":"26617538-bc30-43ce-86d6-5bb08ad00551","Type":"ContainerDied","Data":"1948222b58f2515b0eadb3f2c8c93b0930b191fa7717a3321166eb90e239f216"}
Sep 17 23:53:27.978026 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:27.978004    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:53:28.079531 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.079472    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-kube-rbac-proxy-sar-config\") pod \"26617538-bc30-43ce-86d6-5bb08ad00551\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") "
Sep 17 23:53:28.079531 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.079507    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26617538-bc30-43ce-86d6-5bb08ad00551-proxy-tls\") pod \"26617538-bc30-43ce-86d6-5bb08ad00551\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") "
Sep 17 23:53:28.079704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.079639    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8rmx\" (UniqueName: \"kubernetes.io/projected/26617538-bc30-43ce-86d6-5bb08ad00551-kube-api-access-m8rmx\") pod \"26617538-bc30-43ce-86d6-5bb08ad00551\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") "
Sep 17 23:53:28.079704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.079686    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26617538-bc30-43ce-86d6-5bb08ad00551-kserve-provision-location\") pod \"26617538-bc30-43ce-86d6-5bb08ad00551\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") "
Sep 17 23:53:28.079818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.079718    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-cabundle-cert\") pod \"26617538-bc30-43ce-86d6-5bb08ad00551\" (UID: \"26617538-bc30-43ce-86d6-5bb08ad00551\") "
Sep 17 23:53:28.079818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.079767    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26617538-bc30-43ce-86d6-5bb08ad00551" (UID: "26617538-bc30-43ce-86d6-5bb08ad00551"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:53:28.080079 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.080018    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26617538-bc30-43ce-86d6-5bb08ad00551-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26617538-bc30-43ce-86d6-5bb08ad00551" (UID: "26617538-bc30-43ce-86d6-5bb08ad00551"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:53:28.080079 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.080067    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:53:28.080243 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.080181    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26617538-bc30-43ce-86d6-5bb08ad00551" (UID: "26617538-bc30-43ce-86d6-5bb08ad00551"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:53:28.082309 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.082289    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26617538-bc30-43ce-86d6-5bb08ad00551-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26617538-bc30-43ce-86d6-5bb08ad00551" (UID: "26617538-bc30-43ce-86d6-5bb08ad00551"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:53:28.082309 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.082289    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26617538-bc30-43ce-86d6-5bb08ad00551-kube-api-access-m8rmx" (OuterVolumeSpecName: "kube-api-access-m8rmx") pod "26617538-bc30-43ce-86d6-5bb08ad00551" (UID: "26617538-bc30-43ce-86d6-5bb08ad00551"). InnerVolumeSpecName "kube-api-access-m8rmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:53:28.180897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.180865    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26617538-bc30-43ce-86d6-5bb08ad00551-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:53:28.180897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.180895    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8rmx\" (UniqueName: \"kubernetes.io/projected/26617538-bc30-43ce-86d6-5bb08ad00551-kube-api-access-m8rmx\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:53:28.180897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.180904    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26617538-bc30-43ce-86d6-5bb08ad00551-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:53:28.181053 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.180913    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26617538-bc30-43ce-86d6-5bb08ad00551-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:53:28.972105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.972070    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2" event={"ID":"26617538-bc30-43ce-86d6-5bb08ad00551","Type":"ContainerDied","Data":"0dc0408106fa78b0269436789349e4836655c29543ed783d964c387770f0f142"}
Sep 17 23:53:28.972428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.972109    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"
Sep 17 23:53:28.972428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.972144    2571 scope.go:117] "RemoveContainer" containerID="13671bbc0832701707b16e899c4ba0e9e51a6cd12409eeaad90fbe933be55636"
Sep 17 23:53:28.980179 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.980161    2571 scope.go:117] "RemoveContainer" containerID="1948222b58f2515b0eadb3f2c8c93b0930b191fa7717a3321166eb90e239f216"
Sep 17 23:53:28.988120 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.988103    2571 scope.go:117] "RemoveContainer" containerID="a9a9a31d7b7d86c667ee8705082fa2bf756509c55b05f3abf983a7e6694e00a6"
Sep 17 23:53:28.994543 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.994513    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"]
Sep 17 23:53:28.999013 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:28.998986    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-6gbl2"]
Sep 17 23:53:30.057198 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:30.057171    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" path="/var/lib/kubelet/pods/26617538-bc30-43ce-86d6-5bb08ad00551/volumes"
Sep 17 23:53:32.085314 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:32.085275    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:32.229241 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:32.229210    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:37.971301 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:37.971268    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:42.900955 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:42.900925    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:48.142657 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.142622    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:48.156828 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.156802    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"]
Sep 17 23:53:48.157063 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.157051    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="storage-initializer"
Sep 17 23:53:48.157105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.157065    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="storage-initializer"
Sep 17 23:53:48.157105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.157078    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="kserve-container"
Sep 17 23:53:48.157105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.157083    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="kserve-container"
Sep 17 23:53:48.157105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.157092    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="kube-rbac-proxy"
Sep 17 23:53:48.157105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.157098    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="kube-rbac-proxy"
Sep 17 23:53:48.157260 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.157146    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="kube-rbac-proxy"
Sep 17 23:53:48.157260 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.157153    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="26617538-bc30-43ce-86d6-5bb08ad00551" containerName="kserve-container"
Sep 17 23:53:48.161637 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.161621    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.164238 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.164216    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 17 23:53:48.164423 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.164226    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 23:53:48.164478 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.164470    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:53:48.164524 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.164509    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:53:48.165522 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.165508    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 17 23:53:48.165522 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.165515    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:53:48.169549 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.169526    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"]
Sep 17 23:53:48.220235 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.220210    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e900b5-1036-4522-8372-ffbb155efd6f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.220318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.220243    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.220318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.220260    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e900b5-1036-4522-8372-ffbb155efd6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.220419 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.220354    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pd49\" (UniqueName: \"kubernetes.io/projected/02e900b5-1036-4522-8372-ffbb155efd6f-kube-api-access-6pd49\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.321234 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.321213    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e900b5-1036-4522-8372-ffbb155efd6f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.321348 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.321244    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.321348 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.321261    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e900b5-1036-4522-8372-ffbb155efd6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.321348 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.321290    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pd49\" (UniqueName: \"kubernetes.io/projected/02e900b5-1036-4522-8372-ffbb155efd6f-kube-api-access-6pd49\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.321522 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:53:48.321382    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 23:53:48.321522 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:53:48.321451    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls podName:02e900b5-1036-4522-8372-ffbb155efd6f nodeName:}" failed. No retries permitted until 2026-09-17 23:53:48.82143257 +0000 UTC m=+2553.282350278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls") pod "isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" (UID: "02e900b5-1036-4522-8372-ffbb155efd6f") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 23:53:48.321637 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.321617    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e900b5-1036-4522-8372-ffbb155efd6f-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.321868 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.321853    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e900b5-1036-4522-8372-ffbb155efd6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.330112 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.330095    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pd49\" (UniqueName: \"kubernetes.io/projected/02e900b5-1036-4522-8372-ffbb155efd6f-kube-api-access-6pd49\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.826038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:48.826011    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:48.826154 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:53:48.826109    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 23:53:48.826208 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:53:48.826160    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls podName:02e900b5-1036-4522-8372-ffbb155efd6f nodeName:}" failed. No retries permitted until 2026-09-17 23:53:49.826147323 +0000 UTC m=+2554.287065031 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls") pod "isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" (UID: "02e900b5-1036-4522-8372-ffbb155efd6f") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Sep 17 23:53:49.172687 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:49.172643    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:53:49.833518 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:49.833495    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:49.835669 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:49.835650    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls\") pod \"isvc-sklearn-s3-predictor-8888c8b4b-5wjsv\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:49.971694 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:49.971674    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:50.242633 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:50.242609    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"]
Sep 17 23:53:50.244806 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:53:50.244777    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e900b5_1036_4522_8372_ffbb155efd6f.slice/crio-25d4ea665592ebb757165cf1b9172abf8d75916ab2be97fdeba6e4102d19229c WatchSource:0}: Error finding container 25d4ea665592ebb757165cf1b9172abf8d75916ab2be97fdeba6e4102d19229c: Status 404 returned error can't find the container with id 25d4ea665592ebb757165cf1b9172abf8d75916ab2be97fdeba6e4102d19229c
Sep 17 23:53:51.058381 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:51.058319    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" event={"ID":"02e900b5-1036-4522-8372-ffbb155efd6f","Type":"ContainerStarted","Data":"b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead"}
Sep 17 23:53:51.058583 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:51.058396    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" event={"ID":"02e900b5-1036-4522-8372-ffbb155efd6f","Type":"ContainerStarted","Data":"25d4ea665592ebb757165cf1b9172abf8d75916ab2be97fdeba6e4102d19229c"}
Sep 17 23:53:52.063272 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:52.063247    2571 generic.go:356] "Generic (PLEG): container finished" podID="02e900b5-1036-4522-8372-ffbb155efd6f" containerID="b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead" exitCode=0
Sep 17 23:53:52.063626 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:52.063338    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" event={"ID":"02e900b5-1036-4522-8372-ffbb155efd6f","Type":"ContainerDied","Data":"b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead"}
Sep 17 23:53:53.069850 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:53.069822    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" event={"ID":"02e900b5-1036-4522-8372-ffbb155efd6f","Type":"ContainerStarted","Data":"4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c"}
Sep 17 23:53:53.070201 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:53.069882    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" event={"ID":"02e900b5-1036-4522-8372-ffbb155efd6f","Type":"ContainerStarted","Data":"6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c"}
Sep 17 23:53:53.070201 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:53.070013    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:53.070201 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:53.070043    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:53.071163 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:53.071137    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:53:53.089068 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:53.089026    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podStartSLOduration=4.072129766 podStartE2EDuration="5.089011175s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.016881409s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:53:53.087629364 +0000 UTC m=+2557.548547094" watchObservedRunningTime="2026-09-17 23:53:53.089011175 +0000 UTC m=+2557.549928904"
Sep 17 23:53:54.073277 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:54.073234    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:53:59.078832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:59.078804    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:53:59.079421 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:53:59.079395    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:54:09.080128 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:09.080080    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:54:19.079342 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:19.079287    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:54:29.079370 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:29.079315    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:54:39.079927 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:39.079847    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:54:44.921237 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:44.921205    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:54:49.079532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:49.079492    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:54:52.229619 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:52.229582    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:54:57.348604 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.348569    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:54:57.368899 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.368871    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"]
Sep 17 23:54:57.372419 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.372403    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.374988 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.374968    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 23:54:57.375119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.375003    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 23:54:57.375119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.374965    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:54:57.375119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.375011    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:54:57.381737 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.381715    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"]
Sep 17 23:54:57.407633 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.407612    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b18f8c25-0059-42d2-9d1e-ea891484a10c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.407755 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.407650    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqkrc\" (UniqueName: \"kubernetes.io/projected/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-api-access-tqkrc\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.407755 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.407683    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.407755 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.407708    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.407897 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.407780    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.508162 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.508129    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqkrc\" (UniqueName: \"kubernetes.io/projected/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-api-access-tqkrc\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.508273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.508169    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.508273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.508185    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.508273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.508233    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.508273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.508265    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b18f8c25-0059-42d2-9d1e-ea891484a10c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.508522 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:54:57.508288    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 23:54:57.508522 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:54:57.508372    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls podName:b18f8c25-0059-42d2-9d1e-ea891484a10c nodeName:}" failed. No retries permitted until 2026-09-17 23:54:58.008353018 +0000 UTC m=+2622.469270743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-vgn6s" (UID: "b18f8c25-0059-42d2-9d1e-ea891484a10c") : secret "isvc-paddle-predictor-serving-cert" not found
Sep 17 23:54:57.508639 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.508619    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b18f8c25-0059-42d2-9d1e-ea891484a10c-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.508910 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.508891    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.508910 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.508904    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.517214 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.517196    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqkrc\" (UniqueName: \"kubernetes.io/projected/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-api-access-tqkrc\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:57.556681 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:57.556658    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:54:58.011432 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:58.011406    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:58.013835 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:58.013815    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-vgn6s\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:58.283541 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:58.283494    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:54:58.557629 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:58.557606    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"]
Sep 17 23:54:58.559647 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:54:58.559622    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18f8c25_0059_42d2_9d1e_ea891484a10c.slice/crio-ebf96c33881a55260da8412686d2bb6626c751fd7423f1076a7d9a65759f8e59 WatchSource:0}: Error finding container ebf96c33881a55260da8412686d2bb6626c751fd7423f1076a7d9a65759f8e59: Status 404 returned error can't find the container with id ebf96c33881a55260da8412686d2bb6626c751fd7423f1076a7d9a65759f8e59
Sep 17 23:54:59.080460 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:59.080433    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:54:59.331134 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:59.331052    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" event={"ID":"b18f8c25-0059-42d2-9d1e-ea891484a10c","Type":"ContainerStarted","Data":"2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf"}
Sep 17 23:54:59.331134 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:54:59.331113    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" event={"ID":"b18f8c25-0059-42d2-9d1e-ea891484a10c","Type":"ContainerStarted","Data":"ebf96c33881a55260da8412686d2bb6626c751fd7423f1076a7d9a65759f8e59"}
Sep 17 23:55:03.347616 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:03.347585    2571 generic.go:356] "Generic (PLEG): container finished" podID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerID="2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf" exitCode=0
Sep 17 23:55:03.347995 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:03.347665    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" event={"ID":"b18f8c25-0059-42d2-9d1e-ea891484a10c","Type":"ContainerDied","Data":"2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf"}
Sep 17 23:55:03.348922 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:03.348897    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:55:08.247958 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:08.247927    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"]
Sep 17 23:55:08.248390 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:08.248160    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" containerID="cri-o://6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c" gracePeriod=30
Sep 17 23:55:08.248390 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:08.248219    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kube-rbac-proxy" containerID="cri-o://4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c" gracePeriod=30
Sep 17 23:55:08.277288 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:08.277260    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:55:08.373381 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:08.373348    2571 generic.go:356] "Generic (PLEG): container finished" podID="02e900b5-1036-4522-8372-ffbb155efd6f" containerID="4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c" exitCode=2
Sep 17 23:55:08.373502 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:08.373371    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" event={"ID":"02e900b5-1036-4522-8372-ffbb155efd6f","Type":"ContainerDied","Data":"4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c"}
Sep 17 23:55:08.431888 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:08.431860    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:55:08.682729 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:08.682699    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:55:09.074378 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:09.074289    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 17 23:55:09.079731 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:09.079690    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.52:8080: connect: connection refused"
Sep 17 23:55:14.073887 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:14.073845    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.52:8643/healthz\": dial tcp 10.133.0.52:8643: connect: connection refused"
Sep 17 23:55:14.948606 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:14.948574    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:55:15.053717 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.053696    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e900b5-1036-4522-8372-ffbb155efd6f-kserve-provision-location\") pod \"02e900b5-1036-4522-8372-ffbb155efd6f\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") "
Sep 17 23:55:15.053821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.053756    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e900b5-1036-4522-8372-ffbb155efd6f-kube-rbac-proxy-sar-config\") pod \"02e900b5-1036-4522-8372-ffbb155efd6f\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") "
Sep 17 23:55:15.053821 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.053803    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pd49\" (UniqueName: \"kubernetes.io/projected/02e900b5-1036-4522-8372-ffbb155efd6f-kube-api-access-6pd49\") pod \"02e900b5-1036-4522-8372-ffbb155efd6f\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") "
Sep 17 23:55:15.053946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.053850    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls\") pod \"02e900b5-1036-4522-8372-ffbb155efd6f\" (UID: \"02e900b5-1036-4522-8372-ffbb155efd6f\") "
Sep 17 23:55:15.053997 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.053933    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e900b5-1036-4522-8372-ffbb155efd6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02e900b5-1036-4522-8372-ffbb155efd6f" (UID: "02e900b5-1036-4522-8372-ffbb155efd6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:55:15.054110 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.054085    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02e900b5-1036-4522-8372-ffbb155efd6f-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:55:15.054161 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.054111    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e900b5-1036-4522-8372-ffbb155efd6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02e900b5-1036-4522-8372-ffbb155efd6f" (UID: "02e900b5-1036-4522-8372-ffbb155efd6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:55:15.056105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.056084    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e900b5-1036-4522-8372-ffbb155efd6f-kube-api-access-6pd49" (OuterVolumeSpecName: "kube-api-access-6pd49") pod "02e900b5-1036-4522-8372-ffbb155efd6f" (UID: "02e900b5-1036-4522-8372-ffbb155efd6f"). InnerVolumeSpecName "kube-api-access-6pd49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:55:15.056318 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.056301    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02e900b5-1036-4522-8372-ffbb155efd6f" (UID: "02e900b5-1036-4522-8372-ffbb155efd6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:55:15.155146 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.155118    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02e900b5-1036-4522-8372-ffbb155efd6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:55:15.155411 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.155154    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pd49\" (UniqueName: \"kubernetes.io/projected/02e900b5-1036-4522-8372-ffbb155efd6f-kube-api-access-6pd49\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:55:15.155411 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.155168    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02e900b5-1036-4522-8372-ffbb155efd6f-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:55:15.409673 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.409643    2571 generic.go:356] "Generic (PLEG): container finished" podID="02e900b5-1036-4522-8372-ffbb155efd6f" containerID="6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c" exitCode=0
Sep 17 23:55:15.409836 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.409697    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"
Sep 17 23:55:15.409836 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.409711    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" event={"ID":"02e900b5-1036-4522-8372-ffbb155efd6f","Type":"ContainerDied","Data":"6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c"}
Sep 17 23:55:15.409836 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.409780    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv" event={"ID":"02e900b5-1036-4522-8372-ffbb155efd6f","Type":"ContainerDied","Data":"25d4ea665592ebb757165cf1b9172abf8d75916ab2be97fdeba6e4102d19229c"}
Sep 17 23:55:15.409836 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.409816    2571 scope.go:117] "RemoveContainer" containerID="4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c"
Sep 17 23:55:15.412142 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.412111    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" event={"ID":"b18f8c25-0059-42d2-9d1e-ea891484a10c","Type":"ContainerStarted","Data":"f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7"}
Sep 17 23:55:15.412251 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.412161    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" event={"ID":"b18f8c25-0059-42d2-9d1e-ea891484a10c","Type":"ContainerStarted","Data":"5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb"}
Sep 17 23:55:15.412443 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.412425    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:55:15.412522 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.412484    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:55:15.413813 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.413785    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 23:55:15.418293 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.418277    2571 scope.go:117] "RemoveContainer" containerID="6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c"
Sep 17 23:55:15.425394 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.425379    2571 scope.go:117] "RemoveContainer" containerID="b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead"
Sep 17 23:55:15.432862 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.432847    2571 scope.go:117] "RemoveContainer" containerID="4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c"
Sep 17 23:55:15.433217 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:55:15.433140    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c\": container with ID starting with 4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c not found: ID does not exist" containerID="4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c"
Sep 17 23:55:15.433217 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.433174    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c"} err="failed to get container status \"4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c\": rpc error: code = NotFound desc = could not find container \"4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c\": container with ID starting with 4aa44dbd61f72f6bc94de8a0fd9218fcb236c9085ad5f7b2fd558f5849d7668c not found: ID does not exist"
Sep 17 23:55:15.433217 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.433197    2571 scope.go:117] "RemoveContainer" containerID="6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c"
Sep 17 23:55:15.433685 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:55:15.433643    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c\": container with ID starting with 6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c not found: ID does not exist" containerID="6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c"
Sep 17 23:55:15.433685 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.433672    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c"} err="failed to get container status \"6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c\": rpc error: code = NotFound desc = could not find container \"6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c\": container with ID starting with 6fcf8891929ca6edac6ea28db92743b921ea620ffac292f3fa3e9730ef89e77c not found: ID does not exist"
Sep 17 23:55:15.433829 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.433691    2571 scope.go:117] "RemoveContainer" containerID="b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead"
Sep 17 23:55:15.433998 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:55:15.433979    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead\": container with ID starting with b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead not found: ID does not exist" containerID="b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead"
Sep 17 23:55:15.434042 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.434005    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead"} err="failed to get container status \"b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead\": rpc error: code = NotFound desc = could not find container \"b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead\": container with ID starting with b2e758bda3e46ae44869ab964af396e3f3b976edfcb8ac266e02c90469bc1ead not found: ID does not exist"
Sep 17 23:55:15.435267 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.435225    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podStartSLOduration=2.309110868 podStartE2EDuration="18.435214396s" totalImagesPullingTime="11.695082234s" totalInitContainerRuntime="4.431021294s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:54:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:55:15.433316885 +0000 UTC m=+2639.894234616" watchObservedRunningTime="2026-09-17 23:55:15.435214396 +0000 UTC m=+2639.896132127"
Sep 17 23:55:15.452687 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.449706    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"]
Sep 17 23:55:15.455362 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:15.455339    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-8888c8b4b-5wjsv"]
Sep 17 23:55:16.057932 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:16.057901    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" path="/var/lib/kubelet/pods/02e900b5-1036-4522-8372-ffbb155efd6f/volumes"
Sep 17 23:55:16.416668 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:16.416634    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 23:55:21.421103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:21.421078    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:55:21.421563 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:21.421535    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 23:55:31.421497 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:31.421461    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 23:55:41.422268 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:41.422229    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 23:55:51.422182 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:55:51.422143    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.53:8080: connect: connection refused"
Sep 17 23:56:01.422222 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:01.422195    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:56:08.655071 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:08.654990    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"]
Sep 17 23:56:08.655569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:08.655269    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container" containerID="cri-o://5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb" gracePeriod=30
Sep 17 23:56:08.655569 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:08.655315    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kube-rbac-proxy" containerID="cri-o://f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7" gracePeriod=30
Sep 17 23:56:08.688818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:08.688790    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:09.624091 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:09.624059    2571 generic.go:356] "Generic (PLEG): container finished" podID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerID="f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7" exitCode=2
Sep 17 23:56:09.624259 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:09.624137    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" event={"ID":"b18f8c25-0059-42d2-9d1e-ea891484a10c","Type":"ContainerDied","Data":"f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7"}
Sep 17 23:56:11.092407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.092383    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:56:11.156369 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.156342    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b18f8c25-0059-42d2-9d1e-ea891484a10c-kserve-provision-location\") pod \"b18f8c25-0059-42d2-9d1e-ea891484a10c\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") "
Sep 17 23:56:11.156510 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.156403    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqkrc\" (UniqueName: \"kubernetes.io/projected/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-api-access-tqkrc\") pod \"b18f8c25-0059-42d2-9d1e-ea891484a10c\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") "
Sep 17 23:56:11.156510 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.156429    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls\") pod \"b18f8c25-0059-42d2-9d1e-ea891484a10c\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") "
Sep 17 23:56:11.156510 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.156447    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-rbac-proxy-sar-config\") pod \"b18f8c25-0059-42d2-9d1e-ea891484a10c\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") "
Sep 17 23:56:11.156510 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.156467    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-cabundle-cert\") pod \"b18f8c25-0059-42d2-9d1e-ea891484a10c\" (UID: \"b18f8c25-0059-42d2-9d1e-ea891484a10c\") "
Sep 17 23:56:11.156805 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.156778    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b18f8c25-0059-42d2-9d1e-ea891484a10c" (UID: "b18f8c25-0059-42d2-9d1e-ea891484a10c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:56:11.156900 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.156828    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b18f8c25-0059-42d2-9d1e-ea891484a10c" (UID: "b18f8c25-0059-42d2-9d1e-ea891484a10c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:56:11.158587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.158563    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b18f8c25-0059-42d2-9d1e-ea891484a10c" (UID: "b18f8c25-0059-42d2-9d1e-ea891484a10c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:56:11.158587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.158561    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-api-access-tqkrc" (OuterVolumeSpecName: "kube-api-access-tqkrc") pod "b18f8c25-0059-42d2-9d1e-ea891484a10c" (UID: "b18f8c25-0059-42d2-9d1e-ea891484a10c"). InnerVolumeSpecName "kube-api-access-tqkrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:56:11.164644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.164621    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b18f8c25-0059-42d2-9d1e-ea891484a10c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b18f8c25-0059-42d2-9d1e-ea891484a10c" (UID: "b18f8c25-0059-42d2-9d1e-ea891484a10c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:56:11.257396 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.257343    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18f8c25-0059-42d2-9d1e-ea891484a10c-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:11.257396 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.257363    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:11.257396 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.257373    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b18f8c25-0059-42d2-9d1e-ea891484a10c-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:11.257396 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.257381    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b18f8c25-0059-42d2-9d1e-ea891484a10c-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:11.257396 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.257390    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqkrc\" (UniqueName: \"kubernetes.io/projected/b18f8c25-0059-42d2-9d1e-ea891484a10c-kube-api-access-tqkrc\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:11.634724 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.634694    2571 generic.go:356] "Generic (PLEG): container finished" podID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerID="5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb" exitCode=0
Sep 17 23:56:11.634863 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.634771    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"
Sep 17 23:56:11.634863 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.634774    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" event={"ID":"b18f8c25-0059-42d2-9d1e-ea891484a10c","Type":"ContainerDied","Data":"5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb"}
Sep 17 23:56:11.634863 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.634860    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s" event={"ID":"b18f8c25-0059-42d2-9d1e-ea891484a10c","Type":"ContainerDied","Data":"ebf96c33881a55260da8412686d2bb6626c751fd7423f1076a7d9a65759f8e59"}
Sep 17 23:56:11.634963 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.634887    2571 scope.go:117] "RemoveContainer" containerID="f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7"
Sep 17 23:56:11.643151 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.643137    2571 scope.go:117] "RemoveContainer" containerID="5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb"
Sep 17 23:56:11.650298 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.650283    2571 scope.go:117] "RemoveContainer" containerID="2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf"
Sep 17 23:56:11.657198 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.657174    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"]
Sep 17 23:56:11.657394 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.657378    2571 scope.go:117] "RemoveContainer" containerID="f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7"
Sep 17 23:56:11.657655 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:11.657632    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7\": container with ID starting with f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7 not found: ID does not exist" containerID="f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7"
Sep 17 23:56:11.657745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.657660    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7"} err="failed to get container status \"f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7\": rpc error: code = NotFound desc = could not find container \"f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7\": container with ID starting with f718088be7e2422e848f8b1f7f1f22032381a9203f616ec1758b3c7d0b07f8a7 not found: ID does not exist"
Sep 17 23:56:11.657745 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.657676    2571 scope.go:117] "RemoveContainer" containerID="5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb"
Sep 17 23:56:11.657936 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:11.657919    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb\": container with ID starting with 5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb not found: ID does not exist" containerID="5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb"
Sep 17 23:56:11.658005 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.657946    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb"} err="failed to get container status \"5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb\": rpc error: code = NotFound desc = could not find container \"5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb\": container with ID starting with 5fa00b7505a8fcd3cd54dc02e1c1ad731d2e06d628be07dcd87119518b8abdbb not found: ID does not exist"
Sep 17 23:56:11.658005 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.657968    2571 scope.go:117] "RemoveContainer" containerID="2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf"
Sep 17 23:56:11.658235 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:11.658219    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf\": container with ID starting with 2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf not found: ID does not exist" containerID="2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf"
Sep 17 23:56:11.658288 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.658240    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf"} err="failed to get container status \"2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf\": rpc error: code = NotFound desc = could not find container \"2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf\": container with ID starting with 2020766801d4f7c76a39046bd1631768f3337341b9f86231539fad26c6bc30cf not found: ID does not exist"
Sep 17 23:56:11.661289 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:11.661268    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-vgn6s"]
Sep 17 23:56:12.057544 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:12.057500    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" path="/var/lib/kubelet/pods/b18f8c25-0059-42d2-9d1e-ea891484a10c/volumes"
Sep 17 23:56:13.814818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:13.814787    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:14.165265 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:14.165234    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:14.747038 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:14.747013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:28.474307 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:28.474270    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:29.520340 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.520301    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:29.542892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.542864    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"]
Sep 17 23:56:29.543170 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543159    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="storage-initializer"
Sep 17 23:56:29.543215 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543172    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="storage-initializer"
Sep 17 23:56:29.543215 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543188    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kube-rbac-proxy"
Sep 17 23:56:29.543215 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543194    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kube-rbac-proxy"
Sep 17 23:56:29.543215 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543207    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kube-rbac-proxy"
Sep 17 23:56:29.543215 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543213    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kube-rbac-proxy"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543218    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543223    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543229    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543234    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543239    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="storage-initializer"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543244    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="storage-initializer"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543284    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kserve-container"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543290    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kube-rbac-proxy"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543299    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="02e900b5-1036-4522-8372-ffbb155efd6f" containerName="kube-rbac-proxy"
Sep 17 23:56:29.543395 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.543304    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b18f8c25-0059-42d2-9d1e-ea891484a10c" containerName="kserve-container"
Sep 17 23:56:29.546352 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.546318    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.548831 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.548810    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:56:29.548969 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.548932    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 23:56:29.549173 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.549156    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:56:29.549251 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.549158    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 23:56:29.549251 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.549209    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:56:29.550158 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.550142    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 23:56:29.556646 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.556627    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"]
Sep 17 23:56:29.580652 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.580628    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4edc725-8899-479a-9b30-126a7973d21d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.580735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.580653    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4edc725-8899-479a-9b30-126a7973d21d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.580735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.580675    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4edc725-8899-479a-9b30-126a7973d21d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.580735 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.580700    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplnh\" (UniqueName: \"kubernetes.io/projected/a4edc725-8899-479a-9b30-126a7973d21d-kube-api-access-dplnh\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.641496 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.641476    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:29.681246 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.681222    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4edc725-8899-479a-9b30-126a7973d21d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.681359 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.681249    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4edc725-8899-479a-9b30-126a7973d21d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.681359 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.681269    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4edc725-8899-479a-9b30-126a7973d21d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.681359 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.681292    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplnh\" (UniqueName: \"kubernetes.io/projected/a4edc725-8899-479a-9b30-126a7973d21d-kube-api-access-dplnh\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.681608 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.681592    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4edc725-8899-479a-9b30-126a7973d21d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.682002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.681978    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4edc725-8899-479a-9b30-126a7973d21d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.683752 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.683734    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4edc725-8899-479a-9b30-126a7973d21d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.690351 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.690310    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplnh\" (UniqueName: \"kubernetes.io/projected/a4edc725-8899-479a-9b30-126a7973d21d-kube-api-access-dplnh\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:29.856898 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:29.856876    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:30.128754 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:30.128730    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"]
Sep 17 23:56:30.130743 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:56:30.130720    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4edc725_8899_479a_9b30_126a7973d21d.slice/crio-f29bcb06d7e8cf5d3b0172d5f351fa78746b8978269a67e748091de884e62a33 WatchSource:0}: Error finding container f29bcb06d7e8cf5d3b0172d5f351fa78746b8978269a67e748091de884e62a33: Status 404 returned error can't find the container with id f29bcb06d7e8cf5d3b0172d5f351fa78746b8978269a67e748091de884e62a33
Sep 17 23:56:30.704715 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:30.704677    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r" event={"ID":"a4edc725-8899-479a-9b30-126a7973d21d","Type":"ContainerStarted","Data":"aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231"}
Sep 17 23:56:30.705086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:30.704736    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r" event={"ID":"a4edc725-8899-479a-9b30-126a7973d21d","Type":"ContainerStarted","Data":"f29bcb06d7e8cf5d3b0172d5f351fa78746b8978269a67e748091de884e62a33"}
Sep 17 23:56:34.719648 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:34.719620    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/0.log"
Sep 17 23:56:34.719970 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:34.719656    2571 generic.go:356] "Generic (PLEG): container finished" podID="a4edc725-8899-479a-9b30-126a7973d21d" containerID="aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231" exitCode=1
Sep 17 23:56:34.719970 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:34.719740    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r" event={"ID":"a4edc725-8899-479a-9b30-126a7973d21d","Type":"ContainerDied","Data":"aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231"}
Sep 17 23:56:35.725714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:35.725688    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/0.log"
Sep 17 23:56:35.726054 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:35.725775    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r" event={"ID":"a4edc725-8899-479a-9b30-126a7973d21d","Type":"ContainerStarted","Data":"fa9f00e3508043ba50c0294157078b9b774edab25c80ccf32bf1b3db1b351c9f"}
Sep 17 23:56:37.734358 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:37.734314    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/1.log"
Sep 17 23:56:37.734747 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:37.734732    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/0.log"
Sep 17 23:56:37.734792 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:37.734765    2571 generic.go:356] "Generic (PLEG): container finished" podID="a4edc725-8899-479a-9b30-126a7973d21d" containerID="fa9f00e3508043ba50c0294157078b9b774edab25c80ccf32bf1b3db1b351c9f" exitCode=1
Sep 17 23:56:37.734870 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:37.734846    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r" event={"ID":"a4edc725-8899-479a-9b30-126a7973d21d","Type":"ContainerDied","Data":"fa9f00e3508043ba50c0294157078b9b774edab25c80ccf32bf1b3db1b351c9f"}
Sep 17 23:56:37.734947 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:37.734937    2571 scope.go:117] "RemoveContainer" containerID="aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231"
Sep 17 23:56:37.735302 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:37.735284    2571 scope.go:117] "RemoveContainer" containerID="aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231"
Sep 17 23:56:37.744921 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:37.744898    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_kserve-ci-e2e-test(a4edc725-8899-479a-9b30-126a7973d21d)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r" podUID="a4edc725-8899-479a-9b30-126a7973d21d"
Sep 17 23:56:37.745049 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:37.745027    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_kserve-ci-e2e-test_a4edc725-8899-479a-9b30-126a7973d21d_0 in pod sandbox f29bcb06d7e8cf5d3b0172d5f351fa78746b8978269a67e748091de884e62a33 from index: no such id: 'aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231'" containerID="aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231"
Sep 17 23:56:37.745091 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:37.745060    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_kserve-ci-e2e-test_a4edc725-8899-479a-9b30-126a7973d21d_0 in pod sandbox f29bcb06d7e8cf5d3b0172d5f351fa78746b8978269a67e748091de884e62a33 from index: no such id: 'aee4c2ca958e6f8ecedf0c07f0d35786f6edd792bba0e7a4ae0bd3f79aeab231'"
Sep 17 23:56:37.998369 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:37.998237    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/1.log"
Sep 17 23:56:38.018560 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:38.003465    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/1.log"
Sep 17 23:56:38.018560 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:38.009086    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:56:38.018560 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:38.013886    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 17 23:56:38.739166 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:38.739142    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/1.log"
Sep 17 23:56:39.516230 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.516206    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"]
Sep 17 23:56:39.545512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.545480    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:39.636617 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.636595    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/1.log"
Sep 17 23:56:39.638959 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.638944    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:39.743816 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.743793    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r_a4edc725-8899-479a-9b30-126a7973d21d/storage-initializer/1.log"
Sep 17 23:56:39.744137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.743879    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"
Sep 17 23:56:39.744137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.743918    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r" event={"ID":"a4edc725-8899-479a-9b30-126a7973d21d","Type":"ContainerDied","Data":"f29bcb06d7e8cf5d3b0172d5f351fa78746b8978269a67e748091de884e62a33"}
Sep 17 23:56:39.744137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.743977    2571 scope.go:117] "RemoveContainer" containerID="fa9f00e3508043ba50c0294157078b9b774edab25c80ccf32bf1b3db1b351c9f"
Sep 17 23:56:39.751286 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.751266    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4edc725-8899-479a-9b30-126a7973d21d-kserve-provision-location\") pod \"a4edc725-8899-479a-9b30-126a7973d21d\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") "
Sep 17 23:56:39.751425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.751343    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4edc725-8899-479a-9b30-126a7973d21d-proxy-tls\") pod \"a4edc725-8899-479a-9b30-126a7973d21d\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") "
Sep 17 23:56:39.751425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.751372    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4edc725-8899-479a-9b30-126a7973d21d-kube-rbac-proxy-sar-config\") pod \"a4edc725-8899-479a-9b30-126a7973d21d\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") "
Sep 17 23:56:39.751425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.751389    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dplnh\" (UniqueName: \"kubernetes.io/projected/a4edc725-8899-479a-9b30-126a7973d21d-kube-api-access-dplnh\") pod \"a4edc725-8899-479a-9b30-126a7973d21d\" (UID: \"a4edc725-8899-479a-9b30-126a7973d21d\") "
Sep 17 23:56:39.751599 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.751553    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4edc725-8899-479a-9b30-126a7973d21d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4edc725-8899-479a-9b30-126a7973d21d" (UID: "a4edc725-8899-479a-9b30-126a7973d21d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:56:39.751690 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.751661    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4edc725-8899-479a-9b30-126a7973d21d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4edc725-8899-479a-9b30-126a7973d21d" (UID: "a4edc725-8899-479a-9b30-126a7973d21d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:56:39.753484 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.753465    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4edc725-8899-479a-9b30-126a7973d21d-kube-api-access-dplnh" (OuterVolumeSpecName: "kube-api-access-dplnh") pod "a4edc725-8899-479a-9b30-126a7973d21d" (UID: "a4edc725-8899-479a-9b30-126a7973d21d"). InnerVolumeSpecName "kube-api-access-dplnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:56:39.753560 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.753504    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4edc725-8899-479a-9b30-126a7973d21d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4edc725-8899-479a-9b30-126a7973d21d" (UID: "a4edc725-8899-479a-9b30-126a7973d21d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:56:39.852364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.852346    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4edc725-8899-479a-9b30-126a7973d21d-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:39.852364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.852365    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4edc725-8899-479a-9b30-126a7973d21d-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:39.852491 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.852375    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4edc725-8899-479a-9b30-126a7973d21d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:39.852491 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:39.852385    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dplnh\" (UniqueName: \"kubernetes.io/projected/a4edc725-8899-479a-9b30-126a7973d21d-kube-api-access-dplnh\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:56:40.078334 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.078301    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"]
Sep 17 23:56:40.082753 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.082731    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-556f97dbf8-6f96r"]
Sep 17 23:56:40.657956 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.657925    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"]
Sep 17 23:56:40.658200 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.658189    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4edc725-8899-479a-9b30-126a7973d21d" containerName="storage-initializer"
Sep 17 23:56:40.658254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.658202    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4edc725-8899-479a-9b30-126a7973d21d" containerName="storage-initializer"
Sep 17 23:56:40.658254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.658211    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4edc725-8899-479a-9b30-126a7973d21d" containerName="storage-initializer"
Sep 17 23:56:40.658254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.658216    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4edc725-8899-479a-9b30-126a7973d21d" containerName="storage-initializer"
Sep 17 23:56:40.658372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.658270    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4edc725-8899-479a-9b30-126a7973d21d" containerName="storage-initializer"
Sep 17 23:56:40.658372 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.658278    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4edc725-8899-479a-9b30-126a7973d21d" containerName="storage-initializer"
Sep 17 23:56:40.661266 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.661252    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.664197 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.664167    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 23:56:40.664197 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.664187    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 23:56:40.664420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.664223    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 17 23:56:40.664420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.664173    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 23:56:40.664420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.664295    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 23:56:40.664420 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.664417    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 23:56:40.664614 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.664468    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:56:40.667214 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.667190    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:40.670661 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.670638    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"]
Sep 17 23:56:40.757800 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.757773    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.757800 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.757799    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.758103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.757815    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dff4369-0a78-4069-b2d5-4d82cb46a435-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.758103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.757843    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxqsb\" (UniqueName: \"kubernetes.io/projected/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-api-access-fxqsb\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.758103 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.757921    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.858914 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.858897    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.859022 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.858921    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.859022 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.858938    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dff4369-0a78-4069-b2d5-4d82cb46a435-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.859022 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.858956    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxqsb\" (UniqueName: \"kubernetes.io/projected/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-api-access-fxqsb\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.859190 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:40.859044    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 23:56:40.859190 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.859092    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.859190 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:40.859114    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls podName:0dff4369-0a78-4069-b2d5-4d82cb46a435 nodeName:}" failed. No retries permitted until 2026-09-17 23:56:41.359095812 +0000 UTC m=+2725.820013535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" (UID: "0dff4369-0a78-4069-b2d5-4d82cb46a435") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 23:56:40.859386 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.859349    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dff4369-0a78-4069-b2d5-4d82cb46a435-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.859624 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.859604    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.859662 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.859614    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:40.867782 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:40.867761    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxqsb\" (UniqueName: \"kubernetes.io/projected/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-api-access-fxqsb\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:41.362646 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:41.362623    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:41.362730 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:41.362699    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 23:56:41.362774 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:56:41.362742    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls podName:0dff4369-0a78-4069-b2d5-4d82cb46a435 nodeName:}" failed. No retries permitted until 2026-09-17 23:56:42.362730822 +0000 UTC m=+2726.823648531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" (UID: "0dff4369-0a78-4069-b2d5-4d82cb46a435") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 17 23:56:41.423182 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:41.423159    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:56:42.057946 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:42.057909    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4edc725-8899-479a-9b30-126a7973d21d" path="/var/lib/kubelet/pods/a4edc725-8899-479a-9b30-126a7973d21d/volumes"
Sep 17 23:56:42.370086 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:42.370058    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:42.372461 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:42.372433    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:42.471254 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:42.471224    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:42.745351 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:42.745311    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"]
Sep 17 23:56:42.747795 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:56:42.747755    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dff4369_0a78_4069_b2d5_4d82cb46a435.slice/crio-71697139b411b3c951477a6bc17e0c949bc05468fedbe6fe382aab1a83aeb3b0 WatchSource:0}: Error finding container 71697139b411b3c951477a6bc17e0c949bc05468fedbe6fe382aab1a83aeb3b0: Status 404 returned error can't find the container with id 71697139b411b3c951477a6bc17e0c949bc05468fedbe6fe382aab1a83aeb3b0
Sep 17 23:56:42.763118 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:42.763089    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" event={"ID":"0dff4369-0a78-4069-b2d5-4d82cb46a435","Type":"ContainerStarted","Data":"71697139b411b3c951477a6bc17e0c949bc05468fedbe6fe382aab1a83aeb3b0"}
Sep 17 23:56:43.769832 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:43.769798    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" event={"ID":"0dff4369-0a78-4069-b2d5-4d82cb46a435","Type":"ContainerStarted","Data":"9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb"}
Sep 17 23:56:44.774848 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:44.774819    2571 generic.go:356] "Generic (PLEG): container finished" podID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerID="9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb" exitCode=0
Sep 17 23:56:44.775279 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:44.774869    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" event={"ID":"0dff4369-0a78-4069-b2d5-4d82cb46a435","Type":"ContainerDied","Data":"9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb"}
Sep 17 23:56:45.781085 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:45.781056    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" event={"ID":"0dff4369-0a78-4069-b2d5-4d82cb46a435","Type":"ContainerStarted","Data":"603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34"}
Sep 17 23:56:45.781535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:45.781113    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:45.781535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:45.781137    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" event={"ID":"0dff4369-0a78-4069-b2d5-4d82cb46a435","Type":"ContainerStarted","Data":"53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70"}
Sep 17 23:56:45.781535 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:45.781160    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:45.782215 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:45.782197    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:56:45.799033 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:45.798960    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podStartSLOduration=4.741129174 podStartE2EDuration="5.798943449s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.057814275s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:56:45.79793277 +0000 UTC m=+2730.258850499" watchObservedRunningTime="2026-09-17 23:56:45.798943449 +0000 UTC m=+2730.259861180"
Sep 17 23:56:46.784698 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:46.784659    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:56:51.789216 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:51.789186    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:56:51.789677 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:56:51.789654    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:57:01.790465 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:01.790382    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:57:11.790338 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:11.790288    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:57:21.790290 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:21.790255    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:57:25.221206 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:25.221172    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:57:30.349146 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:30.349111    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:57:30.694791 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:30.694719    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:57:31.789600 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:31.789570    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:57:41.789879 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:41.789802    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:57:51.790497 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:57:51.790470    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:58:00.707119 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:00.707087    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"]
Sep 17 23:58:00.707515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:00.707350    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" containerID="cri-o://53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70" gracePeriod=30
Sep 17 23:58:00.707515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:00.707427    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kube-rbac-proxy" containerID="cri-o://603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34" gracePeriod=30
Sep 17 23:58:00.727732 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:00.727710    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:01.076039 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.075960    2571 generic.go:356] "Generic (PLEG): container finished" podID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerID="603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34" exitCode=2
Sep 17 23:58:01.076175 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.076035    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" event={"ID":"0dff4369-0a78-4069-b2d5-4d82cb46a435","Type":"ContainerDied","Data":"603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34"}
Sep 17 23:58:01.761364 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.761311    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:01.785051 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.784976    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.55:8643/healthz\": dial tcp 10.133.0.55:8643: connect: connection refused"
Sep 17 23:58:01.789658 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.789630    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.55:8080: connect: connection refused"
Sep 17 23:58:01.802890 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.802863    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"]
Sep 17 23:58:01.806251 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.806235    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.808631 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.808614    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 23:58:01.808733 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.808648    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 23:58:01.818741 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.818718    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"]
Sep 17 23:58:01.841458 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.841428    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.841574 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.841464    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.841574 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.841491    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktqk\" (UniqueName: \"kubernetes.io/projected/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-api-access-9ktqk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.841574 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.841549    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.846854 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.846832    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:01.942430 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.942402    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.942564 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.942435    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.942564 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.942472    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktqk\" (UniqueName: \"kubernetes.io/projected/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-api-access-9ktqk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.942564 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.942508    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.942881 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.942860    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.943137 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.943119    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.945126 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.945108    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:01.955261 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:01.955229    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktqk\" (UniqueName: \"kubernetes.io/projected/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-api-access-9ktqk\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:02.116341 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:02.116300    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:02.388704 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:02.388677    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"]
Sep 17 23:58:02.390619 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:58:02.390593    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40817ae0_0ec4_49aa_ba97_91fec6f6cbe6.slice/crio-5f400c014f7e6b58251c4f7410f5bf84160330936f5689063d575d09c8928627 WatchSource:0}: Error finding container 5f400c014f7e6b58251c4f7410f5bf84160330936f5689063d575d09c8928627: Status 404 returned error can't find the container with id 5f400c014f7e6b58251c4f7410f5bf84160330936f5689063d575d09c8928627
Sep 17 23:58:03.092291 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:03.092253    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq" event={"ID":"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6","Type":"ContainerStarted","Data":"a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41"}
Sep 17 23:58:03.092680 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:03.092383    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq" event={"ID":"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6","Type":"ContainerStarted","Data":"5f400c014f7e6b58251c4f7410f5bf84160330936f5689063d575d09c8928627"}
Sep 17 23:58:04.938719 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:04.938696    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:58:05.066247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.066182    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxqsb\" (UniqueName: \"kubernetes.io/projected/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-api-access-fxqsb\") pod \"0dff4369-0a78-4069-b2d5-4d82cb46a435\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") "
Sep 17 23:58:05.066247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.066227    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-rbac-proxy-sar-config\") pod \"0dff4369-0a78-4069-b2d5-4d82cb46a435\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") "
Sep 17 23:58:05.066247 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.066248    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dff4369-0a78-4069-b2d5-4d82cb46a435-kserve-provision-location\") pod \"0dff4369-0a78-4069-b2d5-4d82cb46a435\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") "
Sep 17 23:58:05.066538 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.066268    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls\") pod \"0dff4369-0a78-4069-b2d5-4d82cb46a435\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") "
Sep 17 23:58:05.066538 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.066309    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-cabundle-cert\") pod \"0dff4369-0a78-4069-b2d5-4d82cb46a435\" (UID: \"0dff4369-0a78-4069-b2d5-4d82cb46a435\") "
Sep 17 23:58:05.066653 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.066594    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dff4369-0a78-4069-b2d5-4d82cb46a435-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0dff4369-0a78-4069-b2d5-4d82cb46a435" (UID: "0dff4369-0a78-4069-b2d5-4d82cb46a435"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:58:05.066653 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.066635    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0dff4369-0a78-4069-b2d5-4d82cb46a435" (UID: "0dff4369-0a78-4069-b2d5-4d82cb46a435"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:58:05.066839 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.066808    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0dff4369-0a78-4069-b2d5-4d82cb46a435" (UID: "0dff4369-0a78-4069-b2d5-4d82cb46a435"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:58:05.068577 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.068553    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-api-access-fxqsb" (OuterVolumeSpecName: "kube-api-access-fxqsb") pod "0dff4369-0a78-4069-b2d5-4d82cb46a435" (UID: "0dff4369-0a78-4069-b2d5-4d82cb46a435"). InnerVolumeSpecName "kube-api-access-fxqsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:58:05.068761 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.068735    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0dff4369-0a78-4069-b2d5-4d82cb46a435" (UID: "0dff4369-0a78-4069-b2d5-4d82cb46a435"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:58:05.101859 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.101835    2571 generic.go:356] "Generic (PLEG): container finished" podID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerID="53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70" exitCode=0
Sep 17 23:58:05.101958 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.101920    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" event={"ID":"0dff4369-0a78-4069-b2d5-4d82cb46a435","Type":"ContainerDied","Data":"53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70"}
Sep 17 23:58:05.101958 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.101942    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"
Sep 17 23:58:05.102049 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.101964    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn" event={"ID":"0dff4369-0a78-4069-b2d5-4d82cb46a435","Type":"ContainerDied","Data":"71697139b411b3c951477a6bc17e0c949bc05468fedbe6fe382aab1a83aeb3b0"}
Sep 17 23:58:05.102049 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.101991    2571 scope.go:117] "RemoveContainer" containerID="603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34"
Sep 17 23:58:05.110617 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.110598    2571 scope.go:117] "RemoveContainer" containerID="53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70"
Sep 17 23:58:05.118358 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.118316    2571 scope.go:117] "RemoveContainer" containerID="9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb"
Sep 17 23:58:05.125732 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.125715    2571 scope.go:117] "RemoveContainer" containerID="603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34"
Sep 17 23:58:05.125965 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:58:05.125950    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34\": container with ID starting with 603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34 not found: ID does not exist" containerID="603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34"
Sep 17 23:58:05.126026 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.125971    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34"} err="failed to get container status \"603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34\": rpc error: code = NotFound desc = could not find container \"603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34\": container with ID starting with 603cc7820cf61b63640d6f66d3c749ec1a774b2f8ef8b9e66bc6f6131eecfb34 not found: ID does not exist"
Sep 17 23:58:05.126026 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.125986    2571 scope.go:117] "RemoveContainer" containerID="53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70"
Sep 17 23:58:05.126223 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:58:05.126206    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70\": container with ID starting with 53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70 not found: ID does not exist" containerID="53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70"
Sep 17 23:58:05.126268 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.126229    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70"} err="failed to get container status \"53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70\": rpc error: code = NotFound desc = could not find container \"53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70\": container with ID starting with 53a802d6a70cc2480649e101e3ebee11f1bfd6698f866778a03d162e312f5c70 not found: ID does not exist"
Sep 17 23:58:05.126268 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.126245    2571 scope.go:117] "RemoveContainer" containerID="9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb"
Sep 17 23:58:05.126511 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:58:05.126483    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb\": container with ID starting with 9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb not found: ID does not exist" containerID="9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb"
Sep 17 23:58:05.126560 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.126517    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb"} err="failed to get container status \"9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb\": rpc error: code = NotFound desc = could not find container \"9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb\": container with ID starting with 9056a79980e9b8d41ac43bb7438edb38747b1c249b89daa18d139a1de0c3e8fb not found: ID does not exist"
Sep 17 23:58:05.132827 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.132807    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"]
Sep 17 23:58:05.136213 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.136192    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-b85cc7998-lj4hn"]
Sep 17 23:58:05.167064 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.167037    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxqsb\" (UniqueName: \"kubernetes.io/projected/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-api-access-fxqsb\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:05.167064 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.167062    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:05.167194 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.167077    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dff4369-0a78-4069-b2d5-4d82cb46a435-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:05.167194 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.167090    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dff4369-0a78-4069-b2d5-4d82cb46a435-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:05.167194 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:05.167102    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dff4369-0a78-4069-b2d5-4d82cb46a435-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:06.058161 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:06.058128    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" path="/var/lib/kubelet/pods/0dff4369-0a78-4069-b2d5-4d82cb46a435/volumes"
Sep 17 23:58:06.107425 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:06.107405    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6/storage-initializer/0.log"
Sep 17 23:58:06.107586 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:06.107439    2571 generic.go:356] "Generic (PLEG): container finished" podID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" containerID="a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41" exitCode=1
Sep 17 23:58:06.107586 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:06.107529    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq" event={"ID":"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6","Type":"ContainerDied","Data":"a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41"}
Sep 17 23:58:07.114816 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:07.114789    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6/storage-initializer/0.log"
Sep 17 23:58:07.115189 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:07.114852    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq" event={"ID":"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6","Type":"ContainerStarted","Data":"1c9f7163b6fc2c254691912c0a1544eead74f219f5e2d6acf2aa6835f8508e5c"}
Sep 17 23:58:11.134369 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.134341    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6/storage-initializer/1.log"
Sep 17 23:58:11.134818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.134757    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6/storage-initializer/0.log"
Sep 17 23:58:11.134818 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.134797    2571 generic.go:356] "Generic (PLEG): container finished" podID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" containerID="1c9f7163b6fc2c254691912c0a1544eead74f219f5e2d6acf2aa6835f8508e5c" exitCode=1
Sep 17 23:58:11.134928 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.134872    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq" event={"ID":"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6","Type":"ContainerDied","Data":"1c9f7163b6fc2c254691912c0a1544eead74f219f5e2d6acf2aa6835f8508e5c"}
Sep 17 23:58:11.134981 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.134936    2571 scope.go:117] "RemoveContainer" containerID="a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41"
Sep 17 23:58:11.135285 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.135263    2571 scope.go:117] "RemoveContainer" containerID="a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41"
Sep 17 23:58:11.145876 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:58:11.145853    2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_kserve-ci-e2e-test(40817ae0-0ec4-49aa-ba97-91fec6f6cbe6)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq" podUID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6"
Sep 17 23:58:11.146001 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:58:11.145975    2571 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_kserve-ci-e2e-test_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6_0 in pod sandbox 5f400c014f7e6b58251c4f7410f5bf84160330936f5689063d575d09c8928627 from index: no such id: 'a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41'" containerID="a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41"
Sep 17 23:58:11.146046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.146009    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_kserve-ci-e2e-test_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6_0 in pod sandbox 5f400c014f7e6b58251c4f7410f5bf84160330936f5689063d575d09c8928627 from index: no such id: 'a501c719dff6d18c2a6e9b8d6e0a3e39ba211c2ef76890eeab70584967bd5a41'"
Sep 17 23:58:11.824588 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.824554    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"]
Sep 17 23:58:11.835280 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:11.835257    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:12.140442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.140414    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6/storage-initializer/1.log"
Sep 17 23:58:12.271876 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.271857    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6/storage-initializer/1.log"
Sep 17 23:58:12.275370 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.275354    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:12.321800 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.321773    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kserve-provision-location\") pod \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") "
Sep 17 23:58:12.321957 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.321812    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ktqk\" (UniqueName: \"kubernetes.io/projected/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-api-access-9ktqk\") pod \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") "
Sep 17 23:58:12.321957 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.321842    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-rbac-proxy-sar-config\") pod \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") "
Sep 17 23:58:12.321957 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.321896    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-proxy-tls\") pod \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\" (UID: \"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6\") "
Sep 17 23:58:12.322106 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.322039    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" (UID: "40817ae0-0ec4-49aa-ba97-91fec6f6cbe6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:58:12.322276 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.322246    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" (UID: "40817ae0-0ec4-49aa-ba97-91fec6f6cbe6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:58:12.324126 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.324105    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" (UID: "40817ae0-0ec4-49aa-ba97-91fec6f6cbe6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:58:12.324208 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.324178    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-api-access-9ktqk" (OuterVolumeSpecName: "kube-api-access-9ktqk") pod "40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" (UID: "40817ae0-0ec4-49aa-ba97-91fec6f6cbe6"). InnerVolumeSpecName "kube-api-access-9ktqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:58:12.423002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.422915    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:12.423002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.422943    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ktqk\" (UniqueName: \"kubernetes.io/projected/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-api-access-9ktqk\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:12.423002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.422954    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:12.423002 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.422966    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:58:12.878855 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.878816    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:12.896232 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896199    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"]
Sep 17 23:58:12.896523 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896511    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container"
Sep 17 23:58:12.896565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896525    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container"
Sep 17 23:58:12.896565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896538    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" containerName="storage-initializer"
Sep 17 23:58:12.896565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896544    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" containerName="storage-initializer"
Sep 17 23:58:12.896565 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896560    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="storage-initializer"
Sep 17 23:58:12.896714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896566    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="storage-initializer"
Sep 17 23:58:12.896714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896574    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kube-rbac-proxy"
Sep 17 23:58:12.896714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896579    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kube-rbac-proxy"
Sep 17 23:58:12.896714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896623    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kserve-container"
Sep 17 23:58:12.896714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896632    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" containerName="storage-initializer"
Sep 17 23:58:12.896714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896641    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dff4369-0a78-4069-b2d5-4d82cb46a435" containerName="kube-rbac-proxy"
Sep 17 23:58:12.896714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896681    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" containerName="storage-initializer"
Sep 17 23:58:12.896714 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896686    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" containerName="storage-initializer"
Sep 17 23:58:12.897027 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.896742    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" containerName="storage-initializer"
Sep 17 23:58:12.899784 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.899768    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:12.902695 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.902673    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 23:58:12.902826 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.902676    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:58:12.902826 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.902676    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 23:58:12.909342 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:12.909304    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"]
Sep 17 23:58:13.027554 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.027517    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad05d5fc-a934-493a-8af0-bd935ce713e5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.027554 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.027556    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.027778 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.027576    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.027778 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.027627    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9776\" (UniqueName: \"kubernetes.io/projected/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-api-access-t9776\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.027778 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.027662    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.129007 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.128928    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad05d5fc-a934-493a-8af0-bd935ce713e5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.129007 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.128962    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.129007 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.128981    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.129256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.129097    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9776\" (UniqueName: \"kubernetes.io/projected/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-api-access-t9776\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.129256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.129130    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.129410 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:58:13.129307    2571 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 23:58:13.129470 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.129416    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad05d5fc-a934-493a-8af0-bd935ce713e5-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.129470 ip-10-0-142-246 kubenswrapper[2571]: E0917 23:58:13.129422    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls podName:ad05d5fc-a934-493a-8af0-bd935ce713e5 nodeName:}" failed. No retries permitted until 2026-09-17 23:58:13.629402212 +0000 UTC m=+2818.090319935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" (UID: "ad05d5fc-a934-493a-8af0-bd935ce713e5") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 23:58:13.129671 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.129651    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.129781 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.129762    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.138592 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.138565    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9776\" (UniqueName: \"kubernetes.io/projected/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-api-access-t9776\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.146114 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.146093    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq_40817ae0-0ec4-49aa-ba97-91fec6f6cbe6/storage-initializer/1.log"
Sep 17 23:58:13.146515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.146159    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq" event={"ID":"40817ae0-0ec4-49aa-ba97-91fec6f6cbe6","Type":"ContainerDied","Data":"5f400c014f7e6b58251c4f7410f5bf84160330936f5689063d575d09c8928627"}
Sep 17 23:58:13.146515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.146180    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"
Sep 17 23:58:13.146515 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.146220    2571 scope.go:117] "RemoveContainer" containerID="1c9f7163b6fc2c254691912c0a1544eead74f219f5e2d6acf2aa6835f8508e5c"
Sep 17 23:58:13.184051 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.184021    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"]
Sep 17 23:58:13.189047 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.189011    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-5dcd788598-knrcq"]
Sep 17 23:58:13.281785 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.281753    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:13.632872 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.632845    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.635362 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.635315    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:13.810388 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:13.810354    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:14.058228 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:14.058155    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40817ae0-0ec4-49aa-ba97-91fec6f6cbe6" path="/var/lib/kubelet/pods/40817ae0-0ec4-49aa-ba97-91fec6f6cbe6/volumes"
Sep 17 23:58:14.083798 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:14.083769    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"]
Sep 17 23:58:14.086309 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:58:14.086276    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad05d5fc_a934_493a_8af0_bd935ce713e5.slice/crio-72855e7c631d1ed13c98ba662c412aff329532a145ba0a3a4a0cd2216504f18c WatchSource:0}: Error finding container 72855e7c631d1ed13c98ba662c412aff329532a145ba0a3a4a0cd2216504f18c: Status 404 returned error can't find the container with id 72855e7c631d1ed13c98ba662c412aff329532a145ba0a3a4a0cd2216504f18c
Sep 17 23:58:14.151401 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:14.151364    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" event={"ID":"ad05d5fc-a934-493a-8af0-bd935ce713e5","Type":"ContainerStarted","Data":"38366bc4fe647c96ecbd91698dfd3a07e645de62978de7b6dffb2673bce0ce7e"}
Sep 17 23:58:14.151805 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:14.151447    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" event={"ID":"ad05d5fc-a934-493a-8af0-bd935ce713e5","Type":"ContainerStarted","Data":"72855e7c631d1ed13c98ba662c412aff329532a145ba0a3a4a0cd2216504f18c"}
Sep 17 23:58:16.163943 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:16.163905    2571 generic.go:356] "Generic (PLEG): container finished" podID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerID="38366bc4fe647c96ecbd91698dfd3a07e645de62978de7b6dffb2673bce0ce7e" exitCode=0
Sep 17 23:58:16.164297 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:16.163981    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" event={"ID":"ad05d5fc-a934-493a-8af0-bd935ce713e5","Type":"ContainerDied","Data":"38366bc4fe647c96ecbd91698dfd3a07e645de62978de7b6dffb2673bce0ce7e"}
Sep 17 23:58:17.170428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:17.170393    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" event={"ID":"ad05d5fc-a934-493a-8af0-bd935ce713e5","Type":"ContainerStarted","Data":"78d7a6222e61d6e0f3e996af53e063781c32e1196425d1f371a6cf8a76c4cdfe"}
Sep 17 23:58:17.170808 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:17.170460    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" event={"ID":"ad05d5fc-a934-493a-8af0-bd935ce713e5","Type":"ContainerStarted","Data":"b75a497933a256cccc94fa9b9289f710462e7ca5a38b34d455d5476429630016"}
Sep 17 23:58:17.170808 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:17.170661    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:17.191756 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:17.191703    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podStartSLOduration=4.094339147 podStartE2EDuration="5.191688187s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.09734904s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:58:17.190641227 +0000 UTC m=+2821.651558957" watchObservedRunningTime="2026-09-17 23:58:17.191688187 +0000 UTC m=+2821.652605919"
Sep 17 23:58:18.175081 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:18.175041    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:18.176343 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:18.176300    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:58:19.179466 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:19.179419    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:58:24.184863 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:24.184835    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:58:24.185442 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:24.185409    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:58:32.024426 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:32.024397    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:34.185376 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:34.185317    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:58:37.169512 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.169484    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:37.198477 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.198444    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"]
Sep 17 23:58:37.201867 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.201850    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.204344 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.204307    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 23:58:37.204428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.204319    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 23:58:37.204428 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.204416    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 23:58:37.214601 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.214580    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"]
Sep 17 23:58:37.311726 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.311692    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5a4f56-c791-4da5-9e90-9b076656d3d0-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.311864 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.311735    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.311864 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.311817    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsx7x\" (UniqueName: \"kubernetes.io/projected/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-api-access-fsx7x\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.311951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.311862    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.311951 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.311897    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.412218 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.412184    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.412413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.412227    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.412413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.412376    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5a4f56-c791-4da5-9e90-9b076656d3d0-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.412539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.412437    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.412539 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.412497    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsx7x\" (UniqueName: \"kubernetes.io/projected/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-api-access-fsx7x\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.412644 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.412620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.412936 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.412914    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.413024 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.412962    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.415021 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.414998    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5a4f56-c791-4da5-9e90-9b076656d3d0-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.418043 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.418013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:58:37.421959 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.421905    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsx7x\" (UniqueName: \"kubernetes.io/projected/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-api-access-fsx7x\") pod \"isvc-pmml-predictor-8674795f59-p9rdq\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.511513 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.511487    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:37.785851 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:37.785826    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"]
Sep 17 23:58:37.787874 ip-10-0-142-246 kubenswrapper[2571]: W0917 23:58:37.787847    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5a4f56_c791_4da5_9e90_9b076656d3d0.slice/crio-4baf41ca19c7abf79a799dfbc571370ede61f97cd48e705b274fd915598b2a70 WatchSource:0}: Error finding container 4baf41ca19c7abf79a799dfbc571370ede61f97cd48e705b274fd915598b2a70: Status 404 returned error can't find the container with id 4baf41ca19c7abf79a799dfbc571370ede61f97cd48e705b274fd915598b2a70
Sep 17 23:58:38.255277 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:38.255237    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" event={"ID":"cf5a4f56-c791-4da5-9e90-9b076656d3d0","Type":"ContainerStarted","Data":"843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de"}
Sep 17 23:58:38.255695 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:38.255306    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" event={"ID":"cf5a4f56-c791-4da5-9e90-9b076656d3d0","Type":"ContainerStarted","Data":"4baf41ca19c7abf79a799dfbc571370ede61f97cd48e705b274fd915598b2a70"}
Sep 17 23:58:42.274576 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:42.274479    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerID="843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de" exitCode=0
Sep 17 23:58:42.274576 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:42.274545    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" event={"ID":"cf5a4f56-c791-4da5-9e90-9b076656d3d0","Type":"ContainerDied","Data":"843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de"}
Sep 17 23:58:44.185924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:44.185711    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:58:50.319532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:50.319498    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" event={"ID":"cf5a4f56-c791-4da5-9e90-9b076656d3d0","Type":"ContainerStarted","Data":"03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2"}
Sep 17 23:58:50.319924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:50.319562    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" event={"ID":"cf5a4f56-c791-4da5-9e90-9b076656d3d0","Type":"ContainerStarted","Data":"e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b"}
Sep 17 23:58:50.319924 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:50.319776    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:50.340140 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:50.340094    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podStartSLOduration=2.02788956 podStartE2EDuration="13.340079695s" totalImagesPullingTime="7.315355229s" totalInitContainerRuntime="3.996834906s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:58:50.338013459 +0000 UTC m=+2854.798931218" watchObservedRunningTime="2026-09-17 23:58:50.340079695 +0000 UTC m=+2854.800997426"
Sep 17 23:58:51.324764 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:51.324730    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:51.326256 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:51.326215    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 23:58:52.328545 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:52.328496    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 23:58:54.185563 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:54.185525    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:58:57.332892 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:57.332855    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 17 23:58:57.333476 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:58:57.333452    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 23:59:04.186444 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:04.186403    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:59:07.334105 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:07.334011    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 23:59:10.004444 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:10.004409    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:59:14.185703 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:14.185658    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:59:17.333827 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:17.333784    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 23:59:24.186285 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:24.186249    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.57:8080: connect: connection refused"
Sep 17 23:59:27.334558 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:27.334505    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 23:59:30.058097 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:30.058067    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:59:32.908460 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:32.908421    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"]
Sep 17 23:59:32.908944 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:32.908649    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kube-rbac-proxy" containerID="cri-o://78d7a6222e61d6e0f3e996af53e063781c32e1196425d1f371a6cf8a76c4cdfe" gracePeriod=30
Sep 17 23:59:32.908944 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:32.908633    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container" containerID="cri-o://b75a497933a256cccc94fa9b9289f710462e7ca5a38b34d455d5476429630016" gracePeriod=30
Sep 17 23:59:32.934538 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:32.934512    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:59:33.512743 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:33.512710    2571 generic.go:356] "Generic (PLEG): container finished" podID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerID="78d7a6222e61d6e0f3e996af53e063781c32e1196425d1f371a6cf8a76c4cdfe" exitCode=2
Sep 17 23:59:33.512896 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:33.512778    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" event={"ID":"ad05d5fc-a934-493a-8af0-bd935ce713e5","Type":"ContainerDied","Data":"78d7a6222e61d6e0f3e996af53e063781c32e1196425d1f371a6cf8a76c4cdfe"}
Sep 17 23:59:33.976143 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:33.976108    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:59:34.078532 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:34.078500    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:59:34.180047 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:34.180015    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.57:8643/healthz\": dial tcp 10.133.0.57:8643: connect: connection refused"
Sep 17 23:59:37.334003 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.333961    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 23:59:37.530767 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.530737    2571 generic.go:356] "Generic (PLEG): container finished" podID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerID="b75a497933a256cccc94fa9b9289f710462e7ca5a38b34d455d5476429630016" exitCode=0
Sep 17 23:59:37.530901 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.530812    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" event={"ID":"ad05d5fc-a934-493a-8af0-bd935ce713e5","Type":"ContainerDied","Data":"b75a497933a256cccc94fa9b9289f710462e7ca5a38b34d455d5476429630016"}
Sep 17 23:59:37.551557 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.551538    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:59:37.654872 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.654817    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad05d5fc-a934-493a-8af0-bd935ce713e5-kserve-provision-location\") pod \"ad05d5fc-a934-493a-8af0-bd935ce713e5\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") "
Sep 17 23:59:37.654872 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.654850    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls\") pod \"ad05d5fc-a934-493a-8af0-bd935ce713e5\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") "
Sep 17 23:59:37.655046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.654929    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-cabundle-cert\") pod \"ad05d5fc-a934-493a-8af0-bd935ce713e5\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") "
Sep 17 23:59:37.655046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.654968    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9776\" (UniqueName: \"kubernetes.io/projected/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-api-access-t9776\") pod \"ad05d5fc-a934-493a-8af0-bd935ce713e5\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") "
Sep 17 23:59:37.655046 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.654997    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-rbac-proxy-sar-config\") pod \"ad05d5fc-a934-493a-8af0-bd935ce713e5\" (UID: \"ad05d5fc-a934-493a-8af0-bd935ce713e5\") "
Sep 17 23:59:37.655216 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.655194    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad05d5fc-a934-493a-8af0-bd935ce713e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad05d5fc-a934-493a-8af0-bd935ce713e5" (UID: "ad05d5fc-a934-493a-8af0-bd935ce713e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:59:37.655312 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.655289    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad05d5fc-a934-493a-8af0-bd935ce713e5" (UID: "ad05d5fc-a934-493a-8af0-bd935ce713e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:59:37.655407 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.655389    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad05d5fc-a934-493a-8af0-bd935ce713e5" (UID: "ad05d5fc-a934-493a-8af0-bd935ce713e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:59:37.657191 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.657171    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad05d5fc-a934-493a-8af0-bd935ce713e5" (UID: "ad05d5fc-a934-493a-8af0-bd935ce713e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:59:37.657273 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.657252    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-api-access-t9776" (OuterVolumeSpecName: "kube-api-access-t9776") pod "ad05d5fc-a934-493a-8af0-bd935ce713e5" (UID: "ad05d5fc-a934-493a-8af0-bd935ce713e5"). InnerVolumeSpecName "kube-api-access-t9776". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:59:37.755834 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.755811    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:59:37.755834 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.755832    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9776\" (UniqueName: \"kubernetes.io/projected/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-api-access-t9776\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:59:37.755961 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.755843    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad05d5fc-a934-493a-8af0-bd935ce713e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:59:37.755961 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.755852    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad05d5fc-a934-493a-8af0-bd935ce713e5-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:59:37.755961 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:37.755860    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad05d5fc-a934-493a-8af0-bd935ce713e5-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 17 23:59:38.537552 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:38.537528    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"
Sep 17 23:59:38.538011 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:38.537522    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56" event={"ID":"ad05d5fc-a934-493a-8af0-bd935ce713e5","Type":"ContainerDied","Data":"72855e7c631d1ed13c98ba662c412aff329532a145ba0a3a4a0cd2216504f18c"}
Sep 17 23:59:38.538011 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:38.537670    2571 scope.go:117] "RemoveContainer" containerID="78d7a6222e61d6e0f3e996af53e063781c32e1196425d1f371a6cf8a76c4cdfe"
Sep 17 23:59:38.546171 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:38.546153    2571 scope.go:117] "RemoveContainer" containerID="b75a497933a256cccc94fa9b9289f710462e7ca5a38b34d455d5476429630016"
Sep 17 23:59:38.553413 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:38.553397    2571 scope.go:117] "RemoveContainer" containerID="38366bc4fe647c96ecbd91698dfd3a07e645de62978de7b6dffb2673bce0ce7e"
Sep 17 23:59:38.558298 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:38.558277    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"]
Sep 17 23:59:38.561749 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:38.561731    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-66b557497-cxn56"]
Sep 17 23:59:40.058401 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:40.058364    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" path="/var/lib/kubelet/pods/ad05d5fc-a934-493a-8af0-bd935ce713e5/volumes"
Sep 17 23:59:43.989005 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:43.988959    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:59:45.103587 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:45.103555    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:59:45.342645 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:45.342617    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 17 23:59:47.334346 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:47.334296    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 17 23:59:57.333487 ip-10-0-142-246 kubenswrapper[2571]: I0917 23:59:57.333445    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 18 00:00:07.333392 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:07.333343    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.58:8080: connect: connection refused"
Sep 18 00:00:15.053718 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:15.053679    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 18 00:00:19.338222 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:19.338178    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"]
Sep 18 00:00:19.338751 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:19.338539    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container" containerID="cri-o://e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b" gracePeriod=30
Sep 18 00:00:19.338751 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:19.338562    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kube-rbac-proxy" containerID="cri-o://03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2" gracePeriod=30
Sep 18 00:00:19.348569 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:19.348525    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:00:19.431130 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:19.431105    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:00:19.711698 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:19.711637    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerID="03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2" exitCode=2
Sep 18 00:00:19.711698 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:19.711686    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" event={"ID":"cf5a4f56-c791-4da5-9e90-9b076656d3d0","Type":"ContainerDied","Data":"03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2"}
Sep 18 00:00:22.329262 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:22.329225    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.58:8643/healthz\": dial tcp 10.133.0.58:8643: connect: connection refused"
Sep 18 00:00:23.084598 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.084575    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 18 00:00:23.185316 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.185292    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-rbac-proxy-sar-config\") pod \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") "
Sep 18 00:00:23.185461 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.185344    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-cabundle-cert\") pod \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") "
Sep 18 00:00:23.185461 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.185397    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsx7x\" (UniqueName: \"kubernetes.io/projected/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-api-access-fsx7x\") pod \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") "
Sep 18 00:00:23.185461 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.185421    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5a4f56-c791-4da5-9e90-9b076656d3d0-proxy-tls\") pod \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") "
Sep 18 00:00:23.185461 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.185443    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kserve-provision-location\") pod \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\" (UID: \"cf5a4f56-c791-4da5-9e90-9b076656d3d0\") "
Sep 18 00:00:23.185703 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.185656    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cf5a4f56-c791-4da5-9e90-9b076656d3d0" (UID: "cf5a4f56-c791-4da5-9e90-9b076656d3d0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:00:23.185794 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.185769    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf5a4f56-c791-4da5-9e90-9b076656d3d0" (UID: "cf5a4f56-c791-4da5-9e90-9b076656d3d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:00:23.185844 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.185781    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cf5a4f56-c791-4da5-9e90-9b076656d3d0" (UID: "cf5a4f56-c791-4da5-9e90-9b076656d3d0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:00:23.187615 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.187591    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5a4f56-c791-4da5-9e90-9b076656d3d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cf5a4f56-c791-4da5-9e90-9b076656d3d0" (UID: "cf5a4f56-c791-4da5-9e90-9b076656d3d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:00:23.187711 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.187625    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-api-access-fsx7x" (OuterVolumeSpecName: "kube-api-access-fsx7x") pod "cf5a4f56-c791-4da5-9e90-9b076656d3d0" (UID: "cf5a4f56-c791-4da5-9e90-9b076656d3d0"). InnerVolumeSpecName "kube-api-access-fsx7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:00:23.286413 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.286363    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:00:23.286413 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.286385    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cf5a4f56-c791-4da5-9e90-9b076656d3d0-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:00:23.286413 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.286395    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsx7x\" (UniqueName: \"kubernetes.io/projected/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kube-api-access-fsx7x\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:00:23.286413 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.286403    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cf5a4f56-c791-4da5-9e90-9b076656d3d0-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:00:23.286413 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.286412    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf5a4f56-c791-4da5-9e90-9b076656d3d0-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:00:23.728835 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.728802    2571 generic.go:356] "Generic (PLEG): container finished" podID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerID="e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b" exitCode=0
Sep 18 00:00:23.729191 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.728862    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" event={"ID":"cf5a4f56-c791-4da5-9e90-9b076656d3d0","Type":"ContainerDied","Data":"e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b"}
Sep 18 00:00:23.729191 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.728876    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"
Sep 18 00:00:23.729191 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.728936    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq" event={"ID":"cf5a4f56-c791-4da5-9e90-9b076656d3d0","Type":"ContainerDied","Data":"4baf41ca19c7abf79a799dfbc571370ede61f97cd48e705b274fd915598b2a70"}
Sep 18 00:00:23.729191 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.728964    2571 scope.go:117] "RemoveContainer" containerID="03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2"
Sep 18 00:00:23.738654 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.738635    2571 scope.go:117] "RemoveContainer" containerID="e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b"
Sep 18 00:00:23.746713 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.746693    2571 scope.go:117] "RemoveContainer" containerID="843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de"
Sep 18 00:00:23.752090 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.752062    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"]
Sep 18 00:00:23.754843 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.754821    2571 scope.go:117] "RemoveContainer" containerID="03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2"
Sep 18 00:00:23.755149 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:00:23.755129    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2\": container with ID starting with 03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2 not found: ID does not exist" containerID="03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2"
Sep 18 00:00:23.755219 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.755155    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2"} err="failed to get container status \"03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2\": rpc error: code = NotFound desc = could not find container \"03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2\": container with ID starting with 03e6ac8d084762429895ded05db95ba402e2fbd8e77c9b5635b844f6e6b438d2 not found: ID does not exist"
Sep 18 00:00:23.755219 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.755171    2571 scope.go:117] "RemoveContainer" containerID="e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b"
Sep 18 00:00:23.755452 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:00:23.755434    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b\": container with ID starting with e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b not found: ID does not exist" containerID="e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b"
Sep 18 00:00:23.755504 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.755457    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b"} err="failed to get container status \"e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b\": rpc error: code = NotFound desc = could not find container \"e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b\": container with ID starting with e434d15577030bcbbe71869097ea0965126e8bbbeca36d408e5d53092843d67b not found: ID does not exist"
Sep 18 00:00:23.755504 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.755471    2571 scope.go:117] "RemoveContainer" containerID="843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de"
Sep 18 00:00:23.755785 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:00:23.755763    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de\": container with ID starting with 843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de not found: ID does not exist" containerID="843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de"
Sep 18 00:00:23.755852 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.755793    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de"} err="failed to get container status \"843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de\": rpc error: code = NotFound desc = could not find container \"843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de\": container with ID starting with 843795c8716062da5ec41e1682b591355d2f6516305a761e64ae50261a4f88de not found: ID does not exist"
Sep 18 00:00:23.758255 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:23.758231    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-p9rdq"]
Sep 18 00:00:24.060356 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.060252    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" path="/var/lib/kubelet/pods/cf5a4f56-c791-4da5-9e90-9b076656d3d0/volumes"
Sep 18 00:00:24.512677 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.512645    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:00:24.541919 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.541891    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"]
Sep 18 00:00:24.542195 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542182    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kube-rbac-proxy"
Sep 18 00:00:24.542246 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542197    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kube-rbac-proxy"
Sep 18 00:00:24.542246 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542206    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="storage-initializer"
Sep 18 00:00:24.542246 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542212    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="storage-initializer"
Sep 18 00:00:24.542246 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542221    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container"
Sep 18 00:00:24.542246 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542226    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container"
Sep 18 00:00:24.542246 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542235    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container"
Sep 18 00:00:24.542246 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542241    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container"
Sep 18 00:00:24.542246 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542249    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kube-rbac-proxy"
Sep 18 00:00:24.542554 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542254    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kube-rbac-proxy"
Sep 18 00:00:24.542554 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542266    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="storage-initializer"
Sep 18 00:00:24.542554 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542271    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="storage-initializer"
Sep 18 00:00:24.542554 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542318    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kube-rbac-proxy"
Sep 18 00:00:24.542554 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542345    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad05d5fc-a934-493a-8af0-bd935ce713e5" containerName="kserve-container"
Sep 18 00:00:24.542554 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542356    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kserve-container"
Sep 18 00:00:24.542554 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.542361    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5a4f56-c791-4da5-9e90-9b076656d3d0" containerName="kube-rbac-proxy"
Sep 18 00:00:24.546736 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.546717    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.549862 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.549834    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:00:24.550012 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.549989    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 00:00:24.550060 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.550030    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 00:00:24.551186 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.551171    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 18 00:00:24.551186 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.551180    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 18 00:00:24.551340 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.551185    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 00:00:24.551340 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.551295    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 18 00:00:24.554604 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.554579    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"]
Sep 18 00:00:24.611758 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.611737    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:00:24.696098 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.696075    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.696184 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.696107    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.696184 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.696125    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.696289 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.696202    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwkt\" (UniqueName: \"kubernetes.io/projected/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-api-access-8zwkt\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.696384 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.696296    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.796940 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.796882    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.796940 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.796915    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.796940 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.796934    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.797397 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.796965    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwkt\" (UniqueName: \"kubernetes.io/projected/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-api-access-8zwkt\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.797397 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.796997    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.797397 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.797342    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.797564 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.797547    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.797746 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.797724    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.799639 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.799620    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.806858 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.806823    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwkt\" (UniqueName: \"kubernetes.io/projected/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-api-access-8zwkt\") pod \"isvc-pmml-runtime-predictor-885fc8777-2bxz7\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:24.857695 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:24.857674    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:25.131520 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:25.131491    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"]
Sep 18 00:00:25.133266 ip-10-0-142-246 kubenswrapper[2571]: W0918 00:00:25.133237    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e31dc56_c9a3_4d48_8ba8_4d1fdea44d72.slice/crio-1b8e02ac12415fc446f887d6796f709225c01d8e1451ef66bd92f00ff7fa0143 WatchSource:0}: Error finding container 1b8e02ac12415fc446f887d6796f709225c01d8e1451ef66bd92f00ff7fa0143: Status 404 returned error can't find the container with id 1b8e02ac12415fc446f887d6796f709225c01d8e1451ef66bd92f00ff7fa0143
Sep 18 00:00:25.738991 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:25.738960    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" event={"ID":"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72","Type":"ContainerStarted","Data":"e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36"}
Sep 18 00:00:25.739156 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:25.739016    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" event={"ID":"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72","Type":"ContainerStarted","Data":"1b8e02ac12415fc446f887d6796f709225c01d8e1451ef66bd92f00ff7fa0143"}
Sep 18 00:00:29.758181 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:29.758137    2571 generic.go:356] "Generic (PLEG): container finished" podID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerID="e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36" exitCode=0
Sep 18 00:00:29.758667 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:29.758218    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" event={"ID":"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72","Type":"ContainerDied","Data":"e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36"}
Sep 18 00:00:30.766058 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:30.766012    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" event={"ID":"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72","Type":"ContainerStarted","Data":"d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3"}
Sep 18 00:00:30.766631 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:30.766096    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" event={"ID":"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72","Type":"ContainerStarted","Data":"fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232"}
Sep 18 00:00:30.766631 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:30.766373    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:30.786585 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:30.786530    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podStartSLOduration=3.047836012 podStartE2EDuration="6.786515381s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.738679369s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:00:30.784531135 +0000 UTC m=+2955.245448883" watchObservedRunningTime="2026-09-18 00:00:30.786515381 +0000 UTC m=+2955.247433112"
Sep 18 00:00:31.770284 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:31.770244    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:31.771465 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:31.771436    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:00:32.774095 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:32.774050    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:00:37.778684 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:37.778594    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:00:37.779225 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:37.779190    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:00:47.780076 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:47.780036    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:00:57.779252 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:00:57.779213    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:01:07.779211 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:07.779171    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:01:17.779156 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:17.779112    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:01:26.902564 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:26.902531    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:01:27.779177 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:27.779137    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:01:37.779502 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:37.779447    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:01:38.038901 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:38.038750    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 18 00:01:38.044723 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:38.044705    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 18 00:01:47.779534 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:47.779484    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:01:57.779501 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:01:57.779464    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:02:06.365653 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:06.365617    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"]
Sep 18 00:02:06.366091 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:06.365953    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" containerID="cri-o://fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232" gracePeriod=30
Sep 18 00:02:06.366091 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:06.366022    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kube-rbac-proxy" containerID="cri-o://d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3" gracePeriod=30
Sep 18 00:02:06.386046 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:06.386025    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:02:07.163561 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:07.163523    2571 generic.go:356] "Generic (PLEG): container finished" podID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerID="d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3" exitCode=2
Sep 18 00:02:07.163737 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:07.163562    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" event={"ID":"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72","Type":"ContainerDied","Data":"d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3"}
Sep 18 00:02:07.775238 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:07.775196    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.59:8643/healthz\": dial tcp 10.133.0.59:8643: connect: connection refused"
Sep 18 00:02:07.779523 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:07.779501    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.59:8080: connect: connection refused"
Sep 18 00:02:09.999622 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:09.999598    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:02:10.147421 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.147357    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-rbac-proxy-sar-config\") pod \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") "
Sep 18 00:02:10.147421 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.147395    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zwkt\" (UniqueName: \"kubernetes.io/projected/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-api-access-8zwkt\") pod \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") "
Sep 18 00:02:10.147636 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.147441    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-cabundle-cert\") pod \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") "
Sep 18 00:02:10.147636 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.147461    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-proxy-tls\") pod \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") "
Sep 18 00:02:10.147636 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.147481    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kserve-provision-location\") pod \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\" (UID: \"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72\") "
Sep 18 00:02:10.147865 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.147748    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" (UID: "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:02:10.147988 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.147860    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" (UID: "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:02:10.147988 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.147895    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" (UID: "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:10.149707 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.149684    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" (UID: "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:02:10.149818 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.149791    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-api-access-8zwkt" (OuterVolumeSpecName: "kube-api-access-8zwkt") pod "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" (UID: "6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72"). InnerVolumeSpecName "kube-api-access-8zwkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:02:10.177918 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.177892    2571 generic.go:356] "Generic (PLEG): container finished" podID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerID="fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232" exitCode=0
Sep 18 00:02:10.178024 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.177958    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"
Sep 18 00:02:10.178092 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.177950    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" event={"ID":"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72","Type":"ContainerDied","Data":"fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232"}
Sep 18 00:02:10.178092 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.178077    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7" event={"ID":"6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72","Type":"ContainerDied","Data":"1b8e02ac12415fc446f887d6796f709225c01d8e1451ef66bd92f00ff7fa0143"}
Sep 18 00:02:10.178193 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.178104    2571 scope.go:117] "RemoveContainer" containerID="d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3"
Sep 18 00:02:10.186279 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.186256    2571 scope.go:117] "RemoveContainer" containerID="fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232"
Sep 18 00:02:10.193749 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.193733    2571 scope.go:117] "RemoveContainer" containerID="e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36"
Sep 18 00:02:10.201108 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.201087    2571 scope.go:117] "RemoveContainer" containerID="d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3"
Sep 18 00:02:10.201438 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:02:10.201410    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3\": container with ID starting with d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3 not found: ID does not exist" containerID="d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3"
Sep 18 00:02:10.201526 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.201449    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3"} err="failed to get container status \"d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3\": rpc error: code = NotFound desc = could not find container \"d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3\": container with ID starting with d5b1d4868796d34908265f1f9e8c11378506ef1069160b9f64f3e6529a8908b3 not found: ID does not exist"
Sep 18 00:02:10.201526 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.201472    2571 scope.go:117] "RemoveContainer" containerID="fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232"
Sep 18 00:02:10.201673 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.201658    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"]
Sep 18 00:02:10.201771 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:02:10.201750    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232\": container with ID starting with fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232 not found: ID does not exist" containerID="fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232"
Sep 18 00:02:10.201831 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.201783    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232"} err="failed to get container status \"fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232\": rpc error: code = NotFound desc = could not find container \"fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232\": container with ID starting with fead3bd2bd38aeea9055ec71c8971594a851d85f3a7ebc6370490848d7427232 not found: ID does not exist"
Sep 18 00:02:10.201831 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.201808    2571 scope.go:117] "RemoveContainer" containerID="e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36"
Sep 18 00:02:10.202082 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:02:10.202063    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36\": container with ID starting with e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36 not found: ID does not exist" containerID="e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36"
Sep 18 00:02:10.202141 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.202092    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36"} err="failed to get container status \"e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36\": rpc error: code = NotFound desc = could not find container \"e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36\": container with ID starting with e6252430307ddbe170cd2087888d23c3e2f4172446e60f586ce4368a548c4f36 not found: ID does not exist"
Sep 18 00:02:10.205619 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.205600    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2bxz7"]
Sep 18 00:02:10.248130 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.248106    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:02:10.248130 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.248127    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zwkt\" (UniqueName: \"kubernetes.io/projected/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kube-api-access-8zwkt\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:02:10.248247 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.248136    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:02:10.248247 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.248145    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:02:10.248247 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:10.248152    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:02:11.506757 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.506708    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:02:11.559764 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.559728    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"]
Sep 18 00:02:11.560100 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.560082    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container"
Sep 18 00:02:11.560188 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.560104    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container"
Sep 18 00:02:11.560188 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.560125    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kube-rbac-proxy"
Sep 18 00:02:11.560188 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.560134    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kube-rbac-proxy"
Sep 18 00:02:11.560188 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.560148    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="storage-initializer"
Sep 18 00:02:11.560188 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.560156    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="storage-initializer"
Sep 18 00:02:11.560490 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.560281    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kube-rbac-proxy"
Sep 18 00:02:11.560490 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.560298    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" containerName="kserve-container"
Sep 18 00:02:11.565061 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.565041    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.567568 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.567544    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 00:02:11.567682 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.567578    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-djwfz\""
Sep 18 00:02:11.567932 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.567911    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b53da9-kube-rbac-proxy-sar-config\""
Sep 18 00:02:11.568020 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.567938    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 00:02:11.568837 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.568820    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 00:02:11.568907 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.568844    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-b53da9-predictor-serving-cert\""
Sep 18 00:02:11.568907 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.568870    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:02:11.571804 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.571781    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"]
Sep 18 00:02:11.658469 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.658438    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d5868-795b-469d-951b-3ab652aef627-kserve-provision-location\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.658593 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.658477    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-cabundle-cert\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.658593 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.658521    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwr46\" (UniqueName: \"kubernetes.io/projected/fa0d5868-795b-469d-951b-3ab652aef627-kube-api-access-bwr46\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.658714 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.658604    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d5868-795b-469d-951b-3ab652aef627-proxy-tls\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.658714 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.658641    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.741041 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.740992    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:02:11.759096 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.759049    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d5868-795b-469d-951b-3ab652aef627-proxy-tls\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.759096 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.759081    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.759198 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.759107    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d5868-795b-469d-951b-3ab652aef627-kserve-provision-location\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.759198 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.759131    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-cabundle-cert\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.759198 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.759165    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwr46\" (UniqueName: \"kubernetes.io/projected/fa0d5868-795b-469d-951b-3ab652aef627-kube-api-access-bwr46\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.759524 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.759504    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d5868-795b-469d-951b-3ab652aef627-kserve-provision-location\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.759922 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.759838    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-cabundle-cert\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.759922 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.759850    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-kube-rbac-proxy-sar-config\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.761688 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.761672    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d5868-795b-469d-951b-3ab652aef627-proxy-tls\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.767932 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.767911    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwr46\" (UniqueName: \"kubernetes.io/projected/fa0d5868-795b-469d-951b-3ab652aef627-kube-api-access-bwr46\") pod \"isvc-primary-b53da9-predictor-56898f7c74-jbf5d\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") " pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:11.876744 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:11.876723    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:12.057838 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:12.057773    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72" path="/var/lib/kubelet/pods/6e31dc56-c9a3-4d48-8ba8-4d1fdea44d72/volumes"
Sep 18 00:02:12.149435 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:12.149407    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"]
Sep 18 00:02:12.151561 ip-10-0-142-246 kubenswrapper[2571]: W0918 00:02:12.151533    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0d5868_795b_469d_951b_3ab652aef627.slice/crio-a257cdb71dd9d9b431c7603a0412e7b57c3ee47e39c170013a465a6dc252dcb1 WatchSource:0}: Error finding container a257cdb71dd9d9b431c7603a0412e7b57c3ee47e39c170013a465a6dc252dcb1: Status 404 returned error can't find the container with id a257cdb71dd9d9b431c7603a0412e7b57c3ee47e39c170013a465a6dc252dcb1
Sep 18 00:02:12.189915 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:12.189885    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" event={"ID":"fa0d5868-795b-469d-951b-3ab652aef627","Type":"ContainerStarted","Data":"a257cdb71dd9d9b431c7603a0412e7b57c3ee47e39c170013a465a6dc252dcb1"}
Sep 18 00:02:13.196725 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:13.196682    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" event={"ID":"fa0d5868-795b-469d-951b-3ab652aef627","Type":"ContainerStarted","Data":"6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9"}
Sep 18 00:02:17.214973 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:17.214936    2571 generic.go:356] "Generic (PLEG): container finished" podID="fa0d5868-795b-469d-951b-3ab652aef627" containerID="6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9" exitCode=0
Sep 18 00:02:17.215419 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:17.215011    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" event={"ID":"fa0d5868-795b-469d-951b-3ab652aef627","Type":"ContainerDied","Data":"6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9"}
Sep 18 00:02:18.222034 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:18.221993    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" event={"ID":"fa0d5868-795b-469d-951b-3ab652aef627","Type":"ContainerStarted","Data":"51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a"}
Sep 18 00:02:18.222575 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:18.222067    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" event={"ID":"fa0d5868-795b-469d-951b-3ab652aef627","Type":"ContainerStarted","Data":"8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e"}
Sep 18 00:02:18.222575 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:18.222287    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:18.244941 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:18.244890    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podStartSLOduration=3.180029301 podStartE2EDuration="7.244875679s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.064846378s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:02:18.244394784 +0000 UTC m=+3062.705312514" watchObservedRunningTime="2026-09-18 00:02:18.244875679 +0000 UTC m=+3062.705793410"
Sep 18 00:02:19.226134 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:19.226105    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:19.227414 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:19.227382    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:02:20.231135 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:20.231097    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:02:25.235465 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:25.235437    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:02:25.236056 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:25.236022    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:02:35.236755 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:35.236709    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:02:45.236705 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:45.236663    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:02:55.236573 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:02:55.236531    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:03:05.236393 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:03:05.236318    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:03:15.235911 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:03:15.235870    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:03:22.035501 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:03:22.035466    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:03:25.236579 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:03:25.236533    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:03:35.237031 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:03:35.236997    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:03:56.766863 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:03:56.766792    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:03:56.913691 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:03:56.913664    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:04:14.847515 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:14.847478    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"]
Sep 18 00:04:14.848028 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:14.847708    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" containerID="cri-o://8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e" gracePeriod=30
Sep 18 00:04:14.848028 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:14.847743    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kube-rbac-proxy" containerID="cri-o://51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a" gracePeriod=30
Sep 18 00:04:14.853253 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:14.853151    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:04:14.919079 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:14.919049    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:04:15.232085 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:15.231994    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.60:8643/healthz\": dial tcp 10.133.0.60:8643: connect: connection refused"
Sep 18 00:04:15.236319 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:15.236295    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.60:8080: connect: connection refused"
Sep 18 00:04:15.712157 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:15.712126    2571 generic.go:356] "Generic (PLEG): container finished" podID="fa0d5868-795b-469d-951b-3ab652aef627" containerID="51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a" exitCode=2
Sep 18 00:04:15.712342 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:15.712200    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" event={"ID":"fa0d5868-795b-469d-951b-3ab652aef627","Type":"ContainerDied","Data":"51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a"}
Sep 18 00:04:19.293693 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.293672    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:04:19.481363 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.481254    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwr46\" (UniqueName: \"kubernetes.io/projected/fa0d5868-795b-469d-951b-3ab652aef627-kube-api-access-bwr46\") pod \"fa0d5868-795b-469d-951b-3ab652aef627\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") "
Sep 18 00:04:19.481363 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.481301    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d5868-795b-469d-951b-3ab652aef627-proxy-tls\") pod \"fa0d5868-795b-469d-951b-3ab652aef627\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") "
Sep 18 00:04:19.481363 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.481346    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d5868-795b-469d-951b-3ab652aef627-kserve-provision-location\") pod \"fa0d5868-795b-469d-951b-3ab652aef627\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") "
Sep 18 00:04:19.481642 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.481382    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-cabundle-cert\") pod \"fa0d5868-795b-469d-951b-3ab652aef627\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") "
Sep 18 00:04:19.481642 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.481462    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-kube-rbac-proxy-sar-config\") pod \"fa0d5868-795b-469d-951b-3ab652aef627\" (UID: \"fa0d5868-795b-469d-951b-3ab652aef627\") "
Sep 18 00:04:19.481760 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.481703    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0d5868-795b-469d-951b-3ab652aef627-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa0d5868-795b-469d-951b-3ab652aef627" (UID: "fa0d5868-795b-469d-951b-3ab652aef627"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:04:19.481864 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.481838    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa0d5868-795b-469d-951b-3ab652aef627" (UID: "fa0d5868-795b-469d-951b-3ab652aef627"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:04:19.481915 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.481867    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa0d5868-795b-469d-951b-3ab652aef627" (UID: "fa0d5868-795b-469d-951b-3ab652aef627"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:04:19.483693 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.483671    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0d5868-795b-469d-951b-3ab652aef627-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa0d5868-795b-469d-951b-3ab652aef627" (UID: "fa0d5868-795b-469d-951b-3ab652aef627"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:04:19.483804 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.483714    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0d5868-795b-469d-951b-3ab652aef627-kube-api-access-bwr46" (OuterVolumeSpecName: "kube-api-access-bwr46") pod "fa0d5868-795b-469d-951b-3ab652aef627" (UID: "fa0d5868-795b-469d-951b-3ab652aef627"). InnerVolumeSpecName "kube-api-access-bwr46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:04:19.582132 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.582109    2571 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-cabundle-cert\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:04:19.582132 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.582130    2571 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0d5868-795b-469d-951b-3ab652aef627-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:04:19.582262 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.582139    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwr46\" (UniqueName: \"kubernetes.io/projected/fa0d5868-795b-469d-951b-3ab652aef627-kube-api-access-bwr46\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:04:19.582262 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.582148    2571 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0d5868-795b-469d-951b-3ab652aef627-proxy-tls\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:04:19.582262 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.582156    2571 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0d5868-795b-469d-951b-3ab652aef627-kserve-provision-location\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:04:19.732426 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.732341    2571 generic.go:356] "Generic (PLEG): container finished" podID="fa0d5868-795b-469d-951b-3ab652aef627" containerID="8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e" exitCode=0
Sep 18 00:04:19.732426 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.732355    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" event={"ID":"fa0d5868-795b-469d-951b-3ab652aef627","Type":"ContainerDied","Data":"8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e"}
Sep 18 00:04:19.732426 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.732390    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"
Sep 18 00:04:19.732426 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.732417    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d" event={"ID":"fa0d5868-795b-469d-951b-3ab652aef627","Type":"ContainerDied","Data":"a257cdb71dd9d9b431c7603a0412e7b57c3ee47e39c170013a465a6dc252dcb1"}
Sep 18 00:04:19.732665 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.732438    2571 scope.go:117] "RemoveContainer" containerID="51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a"
Sep 18 00:04:19.741135 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.741119    2571 scope.go:117] "RemoveContainer" containerID="8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e"
Sep 18 00:04:19.748307 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.748291    2571 scope.go:117] "RemoveContainer" containerID="6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9"
Sep 18 00:04:19.755783 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.755760    2571 scope.go:117] "RemoveContainer" containerID="51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a"
Sep 18 00:04:19.755889 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.755874    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"]
Sep 18 00:04:19.756089 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:04:19.756073    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a\": container with ID starting with 51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a not found: ID does not exist" containerID="51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a"
Sep 18 00:04:19.756130 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.756098    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a"} err="failed to get container status \"51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a\": rpc error: code = NotFound desc = could not find container \"51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a\": container with ID starting with 51eaeb03becc488a5eae4494bafa16fc42710ded01e7bae7ebb773908e7a7c9a not found: ID does not exist"
Sep 18 00:04:19.756130 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.756116    2571 scope.go:117] "RemoveContainer" containerID="8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e"
Sep 18 00:04:19.756397 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:04:19.756377    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e\": container with ID starting with 8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e not found: ID does not exist" containerID="8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e"
Sep 18 00:04:19.756486 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.756403    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e"} err="failed to get container status \"8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e\": rpc error: code = NotFound desc = could not find container \"8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e\": container with ID starting with 8226e196e3dd1d21a5fae7b463d65c13bc57c3d3bf2dc1eb3e788d9b0e60469e not found: ID does not exist"
Sep 18 00:04:19.756486 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.756418    2571 scope.go:117] "RemoveContainer" containerID="6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9"
Sep 18 00:04:19.756690 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:04:19.756676    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9\": container with ID starting with 6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9 not found: ID does not exist" containerID="6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9"
Sep 18 00:04:19.756736 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.756695    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9"} err="failed to get container status \"6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9\": rpc error: code = NotFound desc = could not find container \"6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9\": container with ID starting with 6478a683e6632c78262d756c4a98d3dcc85a998446d06bf0b8d7d9e20d86d0b9 not found: ID does not exist"
Sep 18 00:04:19.760069 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:19.760048    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-b53da9-predictor-56898f7c74-jbf5d"]
Sep 18 00:04:20.058263 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:20.058182    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0d5868-795b-469d-951b-3ab652aef627" path="/var/lib/kubelet/pods/fa0d5868-795b-469d-951b-3ab652aef627/volumes"
Sep 18 00:04:21.522056 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522025    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hb59h/must-gather-nrbfz"]
Sep 18 00:04:21.522449 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522308    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container"
Sep 18 00:04:21.522449 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522335    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container"
Sep 18 00:04:21.522449 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522345    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kube-rbac-proxy"
Sep 18 00:04:21.522449 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522350    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kube-rbac-proxy"
Sep 18 00:04:21.522449 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522360    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="storage-initializer"
Sep 18 00:04:21.522449 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522366    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="storage-initializer"
Sep 18 00:04:21.522449 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522418    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kserve-container"
Sep 18 00:04:21.522449 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.522425    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0d5868-795b-469d-951b-3ab652aef627" containerName="kube-rbac-proxy"
Sep 18 00:04:21.526862 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.526843    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:04:21.533757 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.533733    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hb59h\"/\"openshift-service-ca.crt\""
Sep 18 00:04:21.533908 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.533767    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hb59h\"/\"kube-root-ca.crt\""
Sep 18 00:04:21.538370 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.538350    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hb59h/must-gather-nrbfz"]
Sep 18 00:04:21.545141 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.545121    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:04:21.595296 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.595273    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38b25c20-f22c-43d3-ae59-962a9736cefc-must-gather-output\") pod \"must-gather-nrbfz\" (UID: \"38b25c20-f22c-43d3-ae59-962a9736cefc\") " pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:04:21.595428 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.595350    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gft2\" (UniqueName: \"kubernetes.io/projected/38b25c20-f22c-43d3-ae59-962a9736cefc-kube-api-access-8gft2\") pod \"must-gather-nrbfz\" (UID: \"38b25c20-f22c-43d3-ae59-962a9736cefc\") " pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:04:21.620036 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.620013    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:04:21.696253 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.696228    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gft2\" (UniqueName: \"kubernetes.io/projected/38b25c20-f22c-43d3-ae59-962a9736cefc-kube-api-access-8gft2\") pod \"must-gather-nrbfz\" (UID: \"38b25c20-f22c-43d3-ae59-962a9736cefc\") " pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:04:21.696374 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.696282    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38b25c20-f22c-43d3-ae59-962a9736cefc-must-gather-output\") pod \"must-gather-nrbfz\" (UID: \"38b25c20-f22c-43d3-ae59-962a9736cefc\") " pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:04:21.696585 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.696566    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38b25c20-f22c-43d3-ae59-962a9736cefc-must-gather-output\") pod \"must-gather-nrbfz\" (UID: \"38b25c20-f22c-43d3-ae59-962a9736cefc\") " pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:04:21.706250 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.706230    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gft2\" (UniqueName: \"kubernetes.io/projected/38b25c20-f22c-43d3-ae59-962a9736cefc-kube-api-access-8gft2\") pod \"must-gather-nrbfz\" (UID: \"38b25c20-f22c-43d3-ae59-962a9736cefc\") " pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:04:21.849644 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:21.849617    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:04:22.122152 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:22.122126    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hb59h/must-gather-nrbfz"]
Sep 18 00:04:22.124627 ip-10-0-142-246 kubenswrapper[2571]: W0918 00:04:22.124600    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38b25c20_f22c_43d3_ae59_962a9736cefc.slice/crio-acec0be40b0b1f83e1622bf838dd721ecb2a625fb7f20fe7b9c13a54c4ebfa0d WatchSource:0}: Error finding container acec0be40b0b1f83e1622bf838dd721ecb2a625fb7f20fe7b9c13a54c4ebfa0d: Status 404 returned error can't find the container with id acec0be40b0b1f83e1622bf838dd721ecb2a625fb7f20fe7b9c13a54c4ebfa0d
Sep 18 00:04:22.126760 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:22.126737    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 00:04:22.745911 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:22.745874    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hb59h/must-gather-nrbfz" event={"ID":"38b25c20-f22c-43d3-ae59-962a9736cefc","Type":"ContainerStarted","Data":"acec0be40b0b1f83e1622bf838dd721ecb2a625fb7f20fe7b9c13a54c4ebfa0d"}
Sep 18 00:04:29.786161 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:29.786120    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hb59h/must-gather-nrbfz" event={"ID":"38b25c20-f22c-43d3-ae59-962a9736cefc","Type":"ContainerStarted","Data":"d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3"}
Sep 18 00:04:29.786588 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:29.786174    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hb59h/must-gather-nrbfz" event={"ID":"38b25c20-f22c-43d3-ae59-962a9736cefc","Type":"ContainerStarted","Data":"03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7"}
Sep 18 00:04:29.803943 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:29.803883    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hb59h/must-gather-nrbfz" podStartSLOduration=2.143058338 podStartE2EDuration="8.803863974s" totalImagesPullingTime="6.660805636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:04:29.801911942 +0000 UTC m=+3194.262829673" watchObservedRunningTime="2026-09-18 00:04:29.803863974 +0000 UTC m=+3194.264781706"
Sep 18 00:04:56.905115 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:56.905072    2571 generic.go:356] "Generic (PLEG): container finished" podID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerID="03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7" exitCode=0
Sep 18 00:04:56.905556 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:56.905152    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hb59h/must-gather-nrbfz" event={"ID":"38b25c20-f22c-43d3-ae59-962a9736cefc","Type":"ContainerDied","Data":"03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7"}
Sep 18 00:04:56.905556 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:56.905511    2571 scope.go:117] "RemoveContainer" containerID="03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7"
Sep 18 00:04:57.306465 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:57.306402    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hb59h_must-gather-nrbfz_38b25c20-f22c-43d3-ae59-962a9736cefc/gather/0.log"
Sep 18 00:04:57.867875 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:57.867841    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:04:58.005766 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.005736    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skn79/must-gather-xlwx2"]
Sep 18 00:04:58.008984 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.008968    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skn79/must-gather-xlwx2"
Sep 18 00:04:58.011631 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.011612    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skn79\"/\"openshift-service-ca.crt\""
Sep 18 00:04:58.011772 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.011662    2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-skn79\"/\"default-dockercfg-q8r7t\""
Sep 18 00:04:58.011772 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.011611    2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-skn79\"/\"kube-root-ca.crt\""
Sep 18 00:04:58.034168 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.034143    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skn79/must-gather-xlwx2"]
Sep 18 00:04:58.041116 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.041093    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:04:58.106602 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.106578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1c04b293-4d1a-403f-99ff-baec7a955100-must-gather-output\") pod \"must-gather-xlwx2\" (UID: \"1c04b293-4d1a-403f-99ff-baec7a955100\") " pod="openshift-must-gather-skn79/must-gather-xlwx2"
Sep 18 00:04:58.106708 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.106616    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9pdj\" (UniqueName: \"kubernetes.io/projected/1c04b293-4d1a-403f-99ff-baec7a955100-kube-api-access-p9pdj\") pod \"must-gather-xlwx2\" (UID: \"1c04b293-4d1a-403f-99ff-baec7a955100\") " pod="openshift-must-gather-skn79/must-gather-xlwx2"
Sep 18 00:04:58.148937 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.148877    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:04:58.207425 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.207399    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pdj\" (UniqueName: \"kubernetes.io/projected/1c04b293-4d1a-403f-99ff-baec7a955100-kube-api-access-p9pdj\") pod \"must-gather-xlwx2\" (UID: \"1c04b293-4d1a-403f-99ff-baec7a955100\") " pod="openshift-must-gather-skn79/must-gather-xlwx2"
Sep 18 00:04:58.207527 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.207470    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1c04b293-4d1a-403f-99ff-baec7a955100-must-gather-output\") pod \"must-gather-xlwx2\" (UID: \"1c04b293-4d1a-403f-99ff-baec7a955100\") " pod="openshift-must-gather-skn79/must-gather-xlwx2"
Sep 18 00:04:58.207804 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.207787    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1c04b293-4d1a-403f-99ff-baec7a955100-must-gather-output\") pod \"must-gather-xlwx2\" (UID: \"1c04b293-4d1a-403f-99ff-baec7a955100\") " pod="openshift-must-gather-skn79/must-gather-xlwx2"
Sep 18 00:04:58.221330 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.221295    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pdj\" (UniqueName: \"kubernetes.io/projected/1c04b293-4d1a-403f-99ff-baec7a955100-kube-api-access-p9pdj\") pod \"must-gather-xlwx2\" (UID: \"1c04b293-4d1a-403f-99ff-baec7a955100\") " pod="openshift-must-gather-skn79/must-gather-xlwx2"
Sep 18 00:04:58.318492 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.318461    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skn79/must-gather-xlwx2"
Sep 18 00:04:58.586406 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.586357    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skn79/must-gather-xlwx2"]
Sep 18 00:04:58.589970 ip-10-0-142-246 kubenswrapper[2571]: W0918 00:04:58.589937    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c04b293_4d1a_403f_99ff_baec7a955100.slice/crio-dcc49bdb7e49390db2d84a7dec90731183ead9abe236b58b47f7b29ae802af8b WatchSource:0}: Error finding container dcc49bdb7e49390db2d84a7dec90731183ead9abe236b58b47f7b29ae802af8b: Status 404 returned error can't find the container with id dcc49bdb7e49390db2d84a7dec90731183ead9abe236b58b47f7b29ae802af8b
Sep 18 00:04:58.914345 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:58.914252    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skn79/must-gather-xlwx2" event={"ID":"1c04b293-4d1a-403f-99ff-baec7a955100","Type":"ContainerStarted","Data":"dcc49bdb7e49390db2d84a7dec90731183ead9abe236b58b47f7b29ae802af8b"}
Sep 18 00:04:59.921632 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:59.921599    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skn79/must-gather-xlwx2" event={"ID":"1c04b293-4d1a-403f-99ff-baec7a955100","Type":"ContainerStarted","Data":"e4e5cccdc61664262d313e01a52977c6bb41832e761eef6a5d3e07ec62dcc2a0"}
Sep 18 00:04:59.922111 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:59.921642    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skn79/must-gather-xlwx2" event={"ID":"1c04b293-4d1a-403f-99ff-baec7a955100","Type":"ContainerStarted","Data":"3262b80351aa2aaf87f8fbbfedbede78fee5d72a0f5b52a2df1fa61e61e67c5d"}
Sep 18 00:04:59.936056 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:04:59.936006    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-skn79/must-gather-xlwx2" podStartSLOduration=2.144897486 podStartE2EDuration="2.935992582s" totalImagesPullingTime="791.095096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:04:59.935346032 +0000 UTC m=+3224.396263766" watchObservedRunningTime="2026-09-18 00:04:59.935992582 +0000 UTC m=+3224.396910313"
Sep 18 00:05:00.878991 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:00.878958    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-46w55_85f4d3fc-3215-4e00-b5e1-175750c70026/global-pull-secret-syncer/0.log"
Sep 18 00:05:01.041695 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:01.041662    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log"
Sep 18 00:05:01.227131 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:01.227034    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-246.ec2.internal_0758ebe72a943e427e2e61837dacf5fc/haproxy/0.log"
Sep 18 00:05:02.850397 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:02.850365    2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-hb59h/must-gather-nrbfz"]
Sep 18 00:05:02.851352 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:02.851308    2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-hb59h/must-gather-nrbfz" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerName="copy" containerID="cri-o://d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3" gracePeriod=2
Sep 18 00:05:02.853262 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:05:02.853220    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrbfz\" is forbidden: User \"system:node:ip-10-0-142-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hb59h\": no relationship found between node 'ip-10-0-142-246.ec2.internal' and this object" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:05:02.854586 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:02.854551    2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-hb59h/must-gather-nrbfz"]
Sep 18 00:05:03.202350 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.200490    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hb59h_must-gather-nrbfz_38b25c20-f22c-43d3-ae59-962a9736cefc/copy/0.log"
Sep 18 00:05:03.202350 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.202252    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:05:03.205368 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:05:03.204500    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrbfz\" is forbidden: User \"system:node:ip-10-0-142-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hb59h\": no relationship found between node 'ip-10-0-142-246.ec2.internal' and this object" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:05:03.355589 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.353546    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38b25c20-f22c-43d3-ae59-962a9736cefc-must-gather-output\") pod \"38b25c20-f22c-43d3-ae59-962a9736cefc\" (UID: \"38b25c20-f22c-43d3-ae59-962a9736cefc\") "
Sep 18 00:05:03.355589 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.353637    2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gft2\" (UniqueName: \"kubernetes.io/projected/38b25c20-f22c-43d3-ae59-962a9736cefc-kube-api-access-8gft2\") pod \"38b25c20-f22c-43d3-ae59-962a9736cefc\" (UID: \"38b25c20-f22c-43d3-ae59-962a9736cefc\") "
Sep 18 00:05:03.357024 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.356950    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38b25c20-f22c-43d3-ae59-962a9736cefc-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "38b25c20-f22c-43d3-ae59-962a9736cefc" (UID: "38b25c20-f22c-43d3-ae59-962a9736cefc"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:05:03.369241 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.369200    2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b25c20-f22c-43d3-ae59-962a9736cefc-kube-api-access-8gft2" (OuterVolumeSpecName: "kube-api-access-8gft2") pod "38b25c20-f22c-43d3-ae59-962a9736cefc" (UID: "38b25c20-f22c-43d3-ae59-962a9736cefc"). InnerVolumeSpecName "kube-api-access-8gft2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:05:03.454712 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.454592    2571 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38b25c20-f22c-43d3-ae59-962a9736cefc-must-gather-output\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:05:03.454712 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.454628    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gft2\" (UniqueName: \"kubernetes.io/projected/38b25c20-f22c-43d3-ae59-962a9736cefc-kube-api-access-8gft2\") on node \"ip-10-0-142-246.ec2.internal\" DevicePath \"\""
Sep 18 00:05:03.652094 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.652030    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:05:03.712258 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.712098    2571 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-629tc_14398bb6-0891-4bde-a292-08c56c4d2341/konnectivity-agent/0.log" limit=1024
Sep 18 00:05:03.947687 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.947653    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-hb59h_must-gather-nrbfz_38b25c20-f22c-43d3-ae59-962a9736cefc/copy/0.log"
Sep 18 00:05:03.948248 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.948087    2571 generic.go:356] "Generic (PLEG): container finished" podID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerID="d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3" exitCode=143
Sep 18 00:05:03.948248 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.948153    2571 scope.go:117] "RemoveContainer" containerID="d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3"
Sep 18 00:05:03.948408 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.948317    2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:05:03.953871 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:05:03.953234    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrbfz\" is forbidden: User \"system:node:ip-10-0-142-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hb59h\": no relationship found between node 'ip-10-0-142-246.ec2.internal' and this object" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:05:03.967587 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:05:03.967516    2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-nrbfz\" is forbidden: User \"system:node:ip-10-0-142-246.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-hb59h\": no relationship found between node 'ip-10-0-142-246.ec2.internal' and this object" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" pod="openshift-must-gather-hb59h/must-gather-nrbfz"
Sep 18 00:05:03.970193 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.969465    2571 scope.go:117] "RemoveContainer" containerID="03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7"
Sep 18 00:05:03.990205 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.990109    2571 scope.go:117] "RemoveContainer" containerID="d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3"
Sep 18 00:05:03.990819 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:05:03.990684    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3\": container with ID starting with d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3 not found: ID does not exist" containerID="d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3"
Sep 18 00:05:03.990819 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.990718    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3"} err="failed to get container status \"d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3\": rpc error: code = NotFound desc = could not find container \"d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3\": container with ID starting with d8ca0bdea30c66390b99952f3cf4cb182328712f7704960822c26a7d7e15c1b3 not found: ID does not exist"
Sep 18 00:05:03.990819 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.990743    2571 scope.go:117] "RemoveContainer" containerID="03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7"
Sep 18 00:05:03.991442 ip-10-0-142-246 kubenswrapper[2571]: E0918 00:05:03.991359    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7\": container with ID starting with 03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7 not found: ID does not exist" containerID="03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7"
Sep 18 00:05:03.991442 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:03.991402    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7"} err="failed to get container status \"03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7\": rpc error: code = NotFound desc = could not find container \"03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7\": container with ID starting with 03709eae52c5b87211be2dc31dc41934c97df4a205282adbcfe51b0d84be68a7 not found: ID does not exist"
Sep 18 00:05:04.062616 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:04.062553    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" path="/var/lib/kubelet/pods/38b25c20-f22c-43d3-ae59-962a9736cefc/volumes"
Sep 18 00:05:05.180984 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.180926    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-n5kx8_74fd4d81-3eb3-43d0-9944-ddc90a0fe735/cluster-monitoring-operator/0.log"
Sep 18 00:05:05.370054 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.370016    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-fspzn_10d39d3d-7f7f-460e-b412-09c0280fdfbd/monitoring-plugin/0.log"
Sep 18 00:05:05.425677 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.425640    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4kngq_5a8f129a-20d8-4df9-ad0a-8018743ec533/node-exporter/0.log"
Sep 18 00:05:05.467026 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.466950    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4kngq_5a8f129a-20d8-4df9-ad0a-8018743ec533/kube-rbac-proxy/0.log"
Sep 18 00:05:05.505347 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.505298    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4kngq_5a8f129a-20d8-4df9-ad0a-8018743ec533/init-textfile/0.log"
Sep 18 00:05:05.827684 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.827648    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8ffd3bc-a89a-4f02-9b68-c4bac041d413/prometheus/0.log"
Sep 18 00:05:05.861359 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.861312    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8ffd3bc-a89a-4f02-9b68-c4bac041d413/config-reloader/0.log"
Sep 18 00:05:05.891017 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.890986    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8ffd3bc-a89a-4f02-9b68-c4bac041d413/thanos-sidecar/0.log"
Sep 18 00:05:05.922273 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.921561    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8ffd3bc-a89a-4f02-9b68-c4bac041d413/kube-rbac-proxy-web/0.log"
Sep 18 00:05:05.949372 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.949312    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8ffd3bc-a89a-4f02-9b68-c4bac041d413/kube-rbac-proxy/0.log"
Sep 18 00:05:05.979221 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:05.979189    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8ffd3bc-a89a-4f02-9b68-c4bac041d413/kube-rbac-proxy-thanos/0.log"
Sep 18 00:05:06.007158 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.007133    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8ffd3bc-a89a-4f02-9b68-c4bac041d413/init-config-reloader/0.log"
Sep 18 00:05:06.046111 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.046086    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-nqc4j_871db9ce-0962-47a2-86d2-6327063f7661/prometheus-operator/0.log"
Sep 18 00:05:06.083644 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.083536    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-nqc4j_871db9ce-0962-47a2-86d2-6327063f7661/kube-rbac-proxy/0.log"
Sep 18 00:05:06.337904 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.337875    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f5b54f8c-vxsjv_32762502-7f12-4775-9465-1c8f9025a694/thanos-query/0.log"
Sep 18 00:05:06.384694 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.384663    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f5b54f8c-vxsjv_32762502-7f12-4775-9465-1c8f9025a694/kube-rbac-proxy-web/0.log"
Sep 18 00:05:06.420544 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.420519    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f5b54f8c-vxsjv_32762502-7f12-4775-9465-1c8f9025a694/kube-rbac-proxy/0.log"
Sep 18 00:05:06.461608 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.461576    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f5b54f8c-vxsjv_32762502-7f12-4775-9465-1c8f9025a694/prom-label-proxy/0.log"
Sep 18 00:05:06.504559 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.504531    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f5b54f8c-vxsjv_32762502-7f12-4775-9465-1c8f9025a694/kube-rbac-proxy-rules/0.log"
Sep 18 00:05:06.554757 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:06.554732    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77f5b54f8c-vxsjv_32762502-7f12-4775-9465-1c8f9025a694/kube-rbac-proxy-metrics/0.log"
Sep 18 00:05:08.387878 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:08.387853    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/2.log"
Sep 18 00:05:08.392828 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:08.392798    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-9w8fn_3b2bd290-c7f4-4784-a732-0cb7a3090b56/console-operator/3.log"
Sep 18 00:05:09.047011 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.046980    2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"]
Sep 18 00:05:09.047300 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.047288    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerName="copy"
Sep 18 00:05:09.047300 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.047301    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerName="copy"
Sep 18 00:05:09.047428 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.047315    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerName="gather"
Sep 18 00:05:09.047428 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.047337    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerName="gather"
Sep 18 00:05:09.047428 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.047405    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerName="gather"
Sep 18 00:05:09.047428 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.047413    2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="38b25c20-f22c-43d3-ae59-962a9736cefc" containerName="copy"
Sep 18 00:05:09.050822 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.050800    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.062839 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.062811    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"]
Sep 18 00:05:09.212734 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.212697    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-proc\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.212901 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.212779    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77qqv\" (UniqueName: \"kubernetes.io/projected/95e82b19-5c14-40b6-a14a-732f5e5da0d7-kube-api-access-77qqv\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.212901 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.212823    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-lib-modules\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.212901 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.212873    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-podres\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.213010 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.212906    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-sys\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313427 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313349    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-proc\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313427 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313417    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77qqv\" (UniqueName: \"kubernetes.io/projected/95e82b19-5c14-40b6-a14a-732f5e5da0d7-kube-api-access-77qqv\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313653 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313466    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-lib-modules\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313653 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313486    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-podres\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313653 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313500    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-proc\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313653 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313511    2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-sys\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313653 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313558    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-sys\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313854 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313658    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-podres\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.313854 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.313662    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95e82b19-5c14-40b6-a14a-732f5e5da0d7-lib-modules\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.323599 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.323574    2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77qqv\" (UniqueName: \"kubernetes.io/projected/95e82b19-5c14-40b6-a14a-732f5e5da0d7-kube-api-access-77qqv\") pod \"perf-node-gather-daemonset-7v4fk\" (UID: \"95e82b19-5c14-40b6-a14a-732f5e5da0d7\") " pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.364598 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.364566    2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:09.641795 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.641758    2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"]
Sep 18 00:05:09.646393 ip-10-0-142-246 kubenswrapper[2571]: W0918 00:05:09.646360    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod95e82b19_5c14_40b6_a14a_732f5e5da0d7.slice/crio-212c1a3d399dcf6710520c897a4c908188f26fabc3bdaadd22e2dfc5b55c6921 WatchSource:0}: Error finding container 212c1a3d399dcf6710520c897a4c908188f26fabc3bdaadd22e2dfc5b55c6921: Status 404 returned error can't find the container with id 212c1a3d399dcf6710520c897a4c908188f26fabc3bdaadd22e2dfc5b55c6921
Sep 18 00:05:09.984906 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.984818    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk" event={"ID":"95e82b19-5c14-40b6-a14a-732f5e5da0d7","Type":"ContainerStarted","Data":"d805e2f701e69542b8039230e29fc1575bb8aa5d68e542148904f45913cd4c16"}
Sep 18 00:05:09.984906 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.984894    2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk" event={"ID":"95e82b19-5c14-40b6-a14a-732f5e5da0d7","Type":"ContainerStarted","Data":"212c1a3d399dcf6710520c897a4c908188f26fabc3bdaadd22e2dfc5b55c6921"}
Sep 18 00:05:09.985143 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:09.985052    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:10.002109 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:10.002040    2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk" podStartSLOduration=1.002020513 podStartE2EDuration="1.002020513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:05:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:05:10.000154386 +0000 UTC m=+3234.461072145" watchObservedRunningTime="2026-09-18 00:05:10.002020513 +0000 UTC m=+3234.462938266"
Sep 18 00:05:10.336353 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:10.336312    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vv4bs_243d5166-07d6-4f35-8e7a-16b00a2961e9/dns/0.log"
Sep 18 00:05:10.359425 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:10.359396    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vv4bs_243d5166-07d6-4f35-8e7a-16b00a2961e9/kube-rbac-proxy/0.log"
Sep 18 00:05:10.494347 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:10.494305    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ztfrq_522d8a18-aacb-42d7-847e-add2e6355433/dns-node-resolver/0.log"
Sep 18 00:05:10.972152 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:10.972120    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-565bb49474-ht2vn_3698b98e-7c45-4f94-9618-857211a5ccf0/registry/0.log"
Sep 18 00:05:11.058016 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:11.057987    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-scwvj_e0cba546-5344-448a-bed8-1252738e58e2/node-ca/0.log"
Sep 18 00:05:12.153844 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:12.153813    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hcfxw_66d22ebd-0199-4fea-a71f-539cafbf0f3d/serve-healthcheck-canary/0.log"
Sep 18 00:05:12.581980 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:12.581879    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42tst_2679bd6e-6fe4-4354-8264-0bffb199bdbd/kube-rbac-proxy/0.log"
Sep 18 00:05:12.605014 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:12.604985    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42tst_2679bd6e-6fe4-4354-8264-0bffb199bdbd/exporter/0.log"
Sep 18 00:05:12.628933 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:12.628908    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-42tst_2679bd6e-6fe4-4354-8264-0bffb199bdbd/extractor/0.log"
Sep 18 00:05:15.657736 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:15.657708    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gs9w9_c7eec705-ebd5-4446-8a0a-0c77945132b8/s3-init/0.log"
Sep 18 00:05:15.677201 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:15.677175    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-gs9w9_c7eec705-ebd5-4446-8a0a-0c77945132b8/download-hf-model/0.log"
Sep 18 00:05:15.737891 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:15.737846    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-mhwkg_7086040b-d590-4c64-b497-d8455053314e/s3-tls-init-custom/0.log"
Sep 18 00:05:15.755647 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:15.755626    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-mhwkg_7086040b-d590-4c64-b497-d8455053314e/download-hf-model/0.log"
Sep 18 00:05:15.786491 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:15.786473    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-84xrw_2e31921b-bb2e-4769-9649-fb77c61c1e68/s3-tls-init-serving/0.log"
Sep 18 00:05:15.805182 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:15.805155    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-84xrw_2e31921b-bb2e-4769-9649-fb77c61c1e68/download-hf-model/0.log"
Sep 18 00:05:15.999646 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:15.999537    2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-skn79/perf-node-gather-daemonset-7v4fk"
Sep 18 00:05:20.243508 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:20.243481    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-8wjfk_b894f5f5-bdbc-4b9d-852f-34566f6321e7/migrator/0.log"
Sep 18 00:05:20.266631 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:20.266601    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-8wjfk_b894f5f5-bdbc-4b9d-852f-34566f6321e7/graceful-termination/0.log"
Sep 18 00:05:20.596259 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:20.596205    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-q89mj_74ed121c-9dd7-441c-b9a5-5e902465c1c0/kube-storage-version-migrator-operator/1.log"
Sep 18 00:05:20.597557 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:20.597535    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-q89mj_74ed121c-9dd7-441c-b9a5-5e902465c1c0/kube-storage-version-migrator-operator/0.log"
Sep 18 00:05:21.439852 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:21.439819    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5w888_14b269cf-41fe-45d2-b1aa-1957dd795540/kube-multus/0.log"
Sep 18 00:05:21.859761 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:21.859731    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xjllt_bb310836-ff74-44fd-9ed2-2c624875cc9b/kube-multus-additional-cni-plugins/0.log"
Sep 18 00:05:21.883689 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:21.883662    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xjllt_bb310836-ff74-44fd-9ed2-2c624875cc9b/egress-router-binary-copy/0.log"
Sep 18 00:05:21.908484 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:21.908462    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xjllt_bb310836-ff74-44fd-9ed2-2c624875cc9b/cni-plugins/0.log"
Sep 18 00:05:21.932610 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:21.932582    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xjllt_bb310836-ff74-44fd-9ed2-2c624875cc9b/bond-cni-plugin/0.log"
Sep 18 00:05:21.964884 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:21.964866    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xjllt_bb310836-ff74-44fd-9ed2-2c624875cc9b/routeoverride-cni/0.log"
Sep 18 00:05:21.989250 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:21.989230    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xjllt_bb310836-ff74-44fd-9ed2-2c624875cc9b/whereabouts-cni-bincopy/0.log"
Sep 18 00:05:22.013603 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:22.013581    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xjllt_bb310836-ff74-44fd-9ed2-2c624875cc9b/whereabouts-cni/0.log"
Sep 18 00:05:22.173596 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:22.173535    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jfgnd_243eb096-f63f-4cb7-8e3a-e82e582b400b/network-metrics-daemon/0.log"
Sep 18 00:05:22.208347 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:22.208297    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jfgnd_243eb096-f63f-4cb7-8e3a-e82e582b400b/kube-rbac-proxy/0.log"
Sep 18 00:05:23.491438 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:23.491404    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jsdb7_4d6e87a1-b303-4d53-ac3e-49acdca61c0d/ovn-controller/0.log"
Sep 18 00:05:23.509427 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:23.509408    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jsdb7_4d6e87a1-b303-4d53-ac3e-49acdca61c0d/ovn-acl-logging/0.log"
Sep 18 00:05:23.536633 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:23.536607    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jsdb7_4d6e87a1-b303-4d53-ac3e-49acdca61c0d/kube-rbac-proxy-node/0.log"
Sep 18 00:05:23.559522 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:23.559506    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jsdb7_4d6e87a1-b303-4d53-ac3e-49acdca61c0d/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 00:05:23.584065 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:23.584049    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jsdb7_4d6e87a1-b303-4d53-ac3e-49acdca61c0d/northd/0.log"
Sep 18 00:05:23.606485 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:23.606470    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jsdb7_4d6e87a1-b303-4d53-ac3e-49acdca61c0d/nbdb/0.log"
Sep 18 00:05:23.630257 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:23.630240    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jsdb7_4d6e87a1-b303-4d53-ac3e-49acdca61c0d/sbdb/0.log"
Sep 18 00:05:23.679149 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:23.679109    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jsdb7_4d6e87a1-b303-4d53-ac3e-49acdca61c0d/ovnkube-controller/0.log"
Sep 18 00:05:24.768592 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:24.768565    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b6w4t_960a9634-3ae7-41f4-83be-a9b9f3d008a7/network-check-target-container/0.log"
Sep 18 00:05:25.753294 ip-10-0-142-246 kubenswrapper[2571]: I0918 00:05:25.753268    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-rfhbd_2600b302-c671-4f21-9817-6bf74a225573/iptables-alerter/0.log"