Sep 18 19:36:11.501463 ip-10-0-141-190 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 19:36:11.501475 ip-10-0-141-190 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 19:36:11.501482 ip-10-0-141-190 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 19:36:11.501636 ip-10-0-141-190 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 19:36:21.510752 ip-10-0-141-190 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 19:36:21.510771 ip-10-0-141-190 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 48dae54da9b6447ea81b10c823c88bb6 --
Sep 18 19:38:27.470329 ip-10-0-141-190 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 19:38:31.229708 ip-10-0-141-190 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:31.229708 ip-10-0-141-190 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 19:38:31.229708 ip-10-0-141-190 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:31.229708 ip-10-0-141-190 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 19:38:31.229708 ip-10-0-141-190 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 19:38:31.230607 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.229786    2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 19:38:31.234890 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234871    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:31.234890 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234888    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234893    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234897    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234901    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234905    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234908    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234911    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234916    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234920    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234926    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234929    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234933    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234936    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234944    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234948    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234951    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234954    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234958    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234961    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:31.234960 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234964    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234967    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234970    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234973    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234976    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234979    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234982    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234986    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234988    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234992    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234995    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.234997    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235000    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235003    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235006    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235009    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235012    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235015    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235017    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235020    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:31.235463 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235023    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235025    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235028    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235030    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235033    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235050    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235052    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235056    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235058    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235061    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235064    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235067    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235070    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235072    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235076    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235079    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235082    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235085    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235088    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235091    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:31.235995 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235093    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235096    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235099    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235102    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235105    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235107    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235110    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235113    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235116    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235119    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235124    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235129    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235132    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235135    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235138    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235141    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235143    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235146    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235149    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235152    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:31.236627 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235156    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235158    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235161    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235164    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235167    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235170    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235173    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235176    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235179    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235182    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235185    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235188    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235191    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235193    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235196    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235199    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235202    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235205    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235208    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235950    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:31.237169 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235959    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235964    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235968    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235971    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235974    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235978    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235982    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235986    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235989    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235992    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235995    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.235997    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236000    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236005    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236009    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236014    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236017    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236020    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:31.237683 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236024    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236028    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236031    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236050    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236053    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236056    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236059    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236062    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236065    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236068    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236071    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236074    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236077    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236081    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236083    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236087    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236089    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236092    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236095    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236098    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:31.238214 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236101    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236103    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236106    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236109    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236112    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236115    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236117    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236120    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236123    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236127    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236130    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236133    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236137    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236140    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236143    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236146    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236149    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236159    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236162    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:31.238746 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236165    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236167    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236170    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236173    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236176    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236179    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236182    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236185    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236188    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236191    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236193    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236196    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236199    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236201    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236204    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236207    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236210    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236212    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236215    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236218    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:31.239298 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236221    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236223    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236227    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236230    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236232    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236237    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236240    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236243    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236245    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236248    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236251    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236253    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236256    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236259    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236262    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236264    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236267    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236270    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236273    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236275    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:31.239824 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.236278    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237072    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237078    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237081    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237084    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237087    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237090    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237093    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237096    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237099    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237102    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237104    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237107    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237111    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237115    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237118    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237121    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237124    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237127    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:31.240372 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237130    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237133    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237137    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237141    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237144    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237147    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237150    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237153    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237156    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237159    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237162    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237165    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237168    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237171    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237174    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237177    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237181    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237184    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237187    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237189    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:31.240886 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237192    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237196    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237199    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237202    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237205    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237208    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237211    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237214    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237217    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237220    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237223    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237225    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237228    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237231    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237234    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237237    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237239    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237243    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237247    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237251    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:31.241421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237254    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237257    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237260    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237263    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237266    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237269    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237272    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237275    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237278    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237281    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237283    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237286    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237289    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237291    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237294    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237297    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237300    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237303    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237305    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237309    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:31.241939 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237312    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237315    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237318    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237321    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237323    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237326    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237329    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237331    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237334    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237337    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237340    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237343    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237346    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237349    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237351    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237354    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237357    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237359    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237362    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:31.242482 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237365    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.237367    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237438    2559 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237482    2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237489    2559 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237493    2559 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237497    2559 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237501    2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237505    2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237510    2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237514    2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237517    2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237521    2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237524    2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237528    2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237532    2559 flags.go:64] FLAG: --cgroup-root=""
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237535    2559 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237538    2559 flags.go:64] FLAG: --client-ca-file=""
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237541    2559 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237545    2559 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237549    2559 flags.go:64] FLAG: --cluster-domain=""
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237552    2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237555    2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237559    2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 19:38:31.242982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237563    2559 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237572    2559 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237575    2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237578    2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237582    2559 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237585    2559 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237588    2559 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237591    2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237595    2559 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237598    2559 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237602    2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237605    2559 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237608    2559 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237611    2559 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237614    2559 flags.go:64] FLAG: --enable-server="true"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237617    2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237621    2559 flags.go:64] FLAG: --event-burst="100"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237624    2559 flags.go:64] FLAG: --event-qps="50"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237628    2559 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237631    2559 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237634    2559 flags.go:64] FLAG: --eviction-hard=""
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237638    2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237641    2559 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237645    2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237650    2559 flags.go:64] FLAG: --eviction-soft=""
Sep 18 19:38:31.243602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237653    2559 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237656    2559 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237659    2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237663    2559 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237666    2559 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237669    2559 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237672    2559 flags.go:64] FLAG: --feature-gates=""
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237676    2559 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237679    2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237683    2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237686    2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237690    2559 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237693    2559 flags.go:64] FLAG: --help="false"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237696    2559 flags.go:64] FLAG: --hostname-override="ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237700    2559 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237703    2559 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237706    2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237710    2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237714    2559 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237717    2559 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237720    2559 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237723    2559 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237726    2559 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 19:38:31.244294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237729    2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237732    2559 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237735    2559 flags.go:64] FLAG: --kube-reserved=""
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237738    2559 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237741    2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237745    2559 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237747    2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237751    2559 flags.go:64] FLAG: --lock-file=""
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237753    2559 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237760    2559 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237763    2559 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237769    2559 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237772    2559 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237775    2559 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237778    2559 flags.go:64] FLAG: --logging-format="text"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237781    2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237784    2559 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237787    2559 flags.go:64] FLAG: --manifest-url=""
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237790    2559 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237795    2559 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237798    2559 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237802    2559 flags.go:64] FLAG: --max-pods="110"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237805    2559 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237808    2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237811    2559 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 19:38:31.245003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237814    2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237818    2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237821    2559 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237828    2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237836    2559 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237839    2559 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237843    2559 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237846    2559 flags.go:64] FLAG: --pod-cidr=""
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237849    2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237854    2559 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237857    2559 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237860    2559 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237863    2559 flags.go:64] FLAG: --port="10250"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237867    2559 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237870    2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-074ec2a49bd770a24"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237873    2559 flags.go:64] FLAG: --qos-reserved=""
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237876    2559 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237880    2559 flags.go:64] FLAG: --register-node="true"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237883    2559 flags.go:64] FLAG: --register-with-taints=""
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237887    2559 flags.go:64] FLAG: --registry-burst="10"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237890    2559 flags.go:64] FLAG: --registry-qps="5"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237893    2559 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237898    2559 flags.go:64] FLAG: --reserved-memory=""
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237902    2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237906    2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 19:38:31.246063 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237909    2559 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237912    2559 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237915    2559 flags.go:64] FLAG: --runonce="false"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237918    2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237922    2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237925    2559 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237929    2559 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237932    2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237935    2559 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237938    2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237942    2559 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237946    2559 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237950    2559 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237953    2559 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237956    2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237959    2559 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237963    2559 flags.go:64] FLAG: --system-cgroups=""
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237966    2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237972    2559 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237975    2559 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237978    2559 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237982    2559 flags.go:64] FLAG: --tls-min-version=""
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237985    2559 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237988    2559 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237991    2559 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 19:38:31.246927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237994    2559 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.237998    2559 flags.go:64] FLAG: --v="2"
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.238002    2559 flags.go:64] FLAG: --version="false"
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.238006    2559 flags.go:64] FLAG: --vmodule=""
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.238011    2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.238015    2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238139    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238144    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238147    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238151    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238155    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238158    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238161    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238165    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238168    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238173    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238177    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238180    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238185    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238189    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:31.247616 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238192    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238196    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238200    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238203    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238206    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238209    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238212    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238215    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238218    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238222    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238225    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238228    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238231    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238234    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238237    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238240    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238245    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238248    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238250    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238253    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:31.248254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238256    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238259    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238262    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238264    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238268    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238270    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238274    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238276    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238279    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238282    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238286    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238289    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238291    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238294    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238297    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238300    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238303    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238305    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238308    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238311    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:31.248787 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238314    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238317    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238320    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238322    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238325    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238328    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238331    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238334    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238338    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238341    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238344    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238347    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238350    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238353    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238357    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238361    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238365    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238368    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238371    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:31.249381 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238374    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238377    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238380    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238384    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238387    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238389    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238392    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238395    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238398    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238401    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238404    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238406    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238409    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238412    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238415    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238417    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238420    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238423    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238426    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238429    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:31.249883 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238432    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238436    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238439    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238442    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238445    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.238448    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.238454    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.247618    2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.247634    2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.247657    2559 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.247664    2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247720    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247725    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247729    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247732    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:31.250421 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247736    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247739    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247742    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247745    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247749    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247752    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247755    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247758    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247761    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247764    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247767    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247770    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247773    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247776    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247779    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247782    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247791    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247794    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247797    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247800    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:31.250821 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247803    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247806    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247809    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247811    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247814    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247817    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247820    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247823    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247826    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247829    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247832    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247835    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247838    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247841    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247844    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247847    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247850    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247853    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247855    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247858    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:31.251369 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247861    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247864    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247867    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247871    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247873    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247877    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247879    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247882    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247885    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247888    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247891    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247895    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247898    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247902    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247905    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247908    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247911    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247914    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247917    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247920    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:31.251954 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247923    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247926    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247929    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247932    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247935    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247937    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247940    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247943    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247946    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247949    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247952    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247955    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247957    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247960    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247963    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247968    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247972    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247975    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247978    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247981    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:31.252529 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247985    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247988    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247991    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247994    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.247997    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248000    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248003    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248006    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248011    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248014    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248018    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248021    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248025    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248027    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:31.253425 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248031    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.248049    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248171    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248177    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248181    2559 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248184    2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248187    2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248191    2559 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248194    2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248197    2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248200    2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248202    2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248205    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248208    2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248212    2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 19:38:31.253965 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248215    2559 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248218    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248221    2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248223    2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248227    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248230    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248233    2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248236    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248239    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248241    2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248244    2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248247    2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248250    2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248253    2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248256    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248259    2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248262    2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248267    2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248272    2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 19:38:31.254395 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248275    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248278    2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248281    2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248284    2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248286    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248289    2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248292    2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248295    2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248298    2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248300    2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248303    2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248306    2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248309    2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248313    2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248316    2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248319    2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248322    2559 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248325    2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248328    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248331    2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 19:38:31.255015 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248334    2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248337    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248339    2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248342    2559 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248345    2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248348    2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248351    2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248354    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248357    2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248359    2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248362    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248365    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248368    2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248371    2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248374    2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248377    2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248381    2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248384    2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248388    2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 19:38:31.255621 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248391    2559 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248394    2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248396    2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248399    2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248402    2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248405    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248407    2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248411    2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248414    2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248417    2559 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248420    2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248423    2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248426    2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248428    2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248431    2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248434    2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248437    2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248440    2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248442    2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 19:38:31.256262 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248447    2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248450    2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248453    2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248457    2559 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248460    2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248463    2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248466    2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248469    2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:31.248472    2559 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.248478    2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.248580    2559 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.251373    2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.252431    2559 server.go:1019] "Starting client certificate rotation"
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.252547    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:31.256837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.252605    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 19:38:31.257356 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.255155    2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:31.257356 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.256994    2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 19:38:31.275683 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.275663    2559 log.go:25] "Validated CRI v1 runtime API"
Sep 18 19:38:31.278556 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.278541    2559 log.go:25] "Validated CRI v1 image API"
Sep 18 19:38:31.279432 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.279417    2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 19:38:31.281723 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.281703    2559 fs.go:135] Filesystem UUIDs: map[3cd02622-2455-4df8-b7d7-16f0703a8bcb:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ba5a5369-1ee5-48d5-ac0f-e89688d439ea:/dev/nvme0n1p3]
Sep 18 19:38:31.281790 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.281721    2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 19:38:31.286398 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.286075    2559 manager.go:216] Machine: {Timestamp:2026-09-18 19:38:31.285572785 +0000 UTC m=+2.248305517 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089407 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28627f71b184294061d9c3625d698d SystemUUID:ec28627f-71b1-8429-4061-d9c3625d698d BootID:48dae54d-a9b6-447e-a81b-10c823c88bb6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c5:16:76:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c5:16:76:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:c4:5e:55:a0:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 19:38:31.286398 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.286388    2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 19:38:31.286601 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.286568    2559 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 19:38:31.288707 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.288476    2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 19:38:31.288856 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.288712    2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 19:38:31.290128 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.290115    2559 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 19:38:31.290188 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.290131    2559 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 19:38:31.290188 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.290140    2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:31.290188 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.290152    2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 19:38:31.291527 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.291513    2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 19:38:31.292696 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.292677    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:31.295369 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295356    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:31.295499 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295488    2559 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 19:38:31.295535 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295521    2559 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 19:38:31.295535 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295533    2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 19:38:31.295593 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295549    2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 19:38:31.295593 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295559    2559 kubelet.go:398] "Adding apiserver pod source"
Sep 18 19:38:31.295593 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295567    2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 19:38:31.295939 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295925    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:31.295983 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.295951    2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 19:38:31.298230 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.298214    2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 19:38:31.300642 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.300628    2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 19:38:31.300688 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.300661    2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 19:38:31.301729 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301716    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 19:38:31.301780 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301734    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 19:38:31.301780 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301740    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 19:38:31.301780 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301746    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 19:38:31.301780 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301756    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 19:38:31.301780 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301762    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 19:38:31.301780 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301776    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 19:38:31.301780 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301782    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 19:38:31.301973 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301789    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 19:38:31.301973 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301794    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 19:38:31.301973 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301803    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 19:38:31.301973 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301808    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 19:38:31.301973 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301835    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 19:38:31.301973 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.301848    2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 19:38:31.305655 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.305640    2559 server.go:1268] "Started kubelet"
Sep 18 19:38:31.305763 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.305715    2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 19:38:31.305821 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.305782    2559 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 19:38:31.305909 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.305877    2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 19:38:31.306504 ip-10-0-141-190 systemd[1]: Started Kubernetes Kubelet.
Sep 18 19:38:31.307094 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.307071    2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 19:38:31.307383 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.307289    2559 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 19:38:31.310238 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.310216    2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 19:38:31.313129 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.313111    2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 19:38:31.313619 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.313597    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 19:38:31.313686 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.313597    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 19:38:31.314737 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.314719    2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 19:38:31.314737 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.314729    2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:31.314904 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.314788    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:31.315369 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.315352    2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 19:38:31.315456 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.315352    2559 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 19:38:31.315456 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.315391    2559 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 19:38:31.315549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.315457    2559 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 19:38:31.315549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.315464    2559 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 19:38:31.316420 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.316405    2559 factory.go:55] Registering systemd factory
Sep 18 19:38:31.316420 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.316419    2559 factory.go:223] Registration of the systemd container factory successfully
Sep 18 19:38:31.316783 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.316724    2559 factory.go:153] Registering CRI-O factory
Sep 18 19:38:31.316783 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.316742    2559 factory.go:223] Registration of the crio container factory successfully
Sep 18 19:38:31.316900 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.316794    2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 19:38:31.316900 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.316812    2559 factory.go:103] Registering Raw factory
Sep 18 19:38:31.316900 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.316828    2559 manager.go:1196] Started watching for new ooms in manager
Sep 18 19:38:31.317327 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.317304    2559 manager.go:349] Starting recovery of all containers
Sep 18 19:38:31.317523 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.317501    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7lh5q"
Sep 18 19:38:31.338387 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.338362    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7lh5q"
Sep 18 19:38:31.342194 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.342159    2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 19:38:31.342320 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.342279    2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 19:38:31.346684 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.345672    2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-190.ec2.internal.18d681854476a97f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-190.ec2.internal,UID:ip-10-0-141-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-190.ec2.internal,},FirstTimestamp:2026-09-18 19:38:31.305611647 +0000 UTC m=+2.268344379,LastTimestamp:2026-09-18 19:38:31.305611647 +0000 UTC m=+2.268344379,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-190.ec2.internal,}"
Sep 18 19:38:31.349444 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.349429    2559 manager.go:354] Recovery completed
Sep 18 19:38:31.353523 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.353508    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:31.355533 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.355517    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:31.355600 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.355547    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:31.355600 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.355559    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:31.356181 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.356167    2559 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 19:38:31.356181 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.356179    2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 19:38:31.356284 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.356180    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:31.356284 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.356200    2559 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 19:38:31.356880 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.356863    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:31.356950 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.356890    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:31.356950 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.356906    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:31.358294 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.358280    2559 policy_none.go:49] "None policy: Start"
Sep 18 19:38:31.358360 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.358299    2559 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 19:38:31.358360 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.358313    2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 19:38:31.360272 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.360258    2559 policy_none.go:47] "Start"
Sep 18 19:38:31.396762 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.396632    2559 manager.go:337] "Starting Device Plugin manager"
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.396802    2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.396813    2559 server.go:87] "Starting device plugin registration server"
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.397060    2559 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.397084    2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.397175    2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.397295    2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.397308    2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.397754    2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 19:38:31.411699 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.397808    2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:31.463046 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.463010    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 19:38:31.464376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.464361    2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 19:38:31.464454 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.464410    2559 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 19:38:31.464454 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.464439    2559 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 19:38:31.464552 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.464480    2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 19:38:31.466950 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.466928    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:31.497457 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.497394    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:31.498317 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.498295    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:31.498400 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.498331    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:31.498400 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.498342    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:31.498400 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.498371    2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.506999 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.506984    2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.507069 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.507005    2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-190.ec2.internal\": node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:31.518765 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.518748    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:31.565325 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.565301    2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal"]
Sep 18 19:38:31.565408 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.565372    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:31.566751 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.566735    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:31.566836 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.566760    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:31.566836 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.566775    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:31.568835 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.568823    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:31.568976 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.568958    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.569053 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.569030    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:31.569498 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.569482    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:31.569561 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.569506    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:31.569561 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.569516    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:31.569637 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.569577    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:31.569637 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.569598    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:31.569637 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.569607    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:31.571721 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.571702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.571809 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.571736    2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 19:38:31.572503 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.572487    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 19:38:31.572591 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.572516    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 19:38:31.572591 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.572529    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeHasSufficientPID"
Sep 18 19:38:31.600692 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.600672    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-190.ec2.internal\" not found" node="ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.605461 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.605445    2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-190.ec2.internal\" not found" node="ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.617367 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.617348    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbcdd6679a6e77f8381bc386784a1377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal\" (UID: \"dbcdd6679a6e77f8381bc386784a1377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.617452 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.617375    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbcdd6679a6e77f8381bc386784a1377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal\" (UID: \"dbcdd6679a6e77f8381bc386784a1377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.617452 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.617392    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c80dc006f68abf1cd999156c6067d05e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-190.ec2.internal\" (UID: \"c80dc006f68abf1cd999156c6067d05e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.619482 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.619461    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:31.717552 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.717518    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbcdd6679a6e77f8381bc386784a1377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal\" (UID: \"dbcdd6679a6e77f8381bc386784a1377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.717552 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.717554    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbcdd6679a6e77f8381bc386784a1377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal\" (UID: \"dbcdd6679a6e77f8381bc386784a1377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.717722 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.717571    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c80dc006f68abf1cd999156c6067d05e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-190.ec2.internal\" (UID: \"c80dc006f68abf1cd999156c6067d05e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.717722 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.717617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c80dc006f68abf1cd999156c6067d05e-config\") pod \"kube-apiserver-proxy-ip-10-0-141-190.ec2.internal\" (UID: \"c80dc006f68abf1cd999156c6067d05e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.717722 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.717653    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbcdd6679a6e77f8381bc386784a1377-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal\" (UID: \"dbcdd6679a6e77f8381bc386784a1377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.717722 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.717671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbcdd6679a6e77f8381bc386784a1377-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal\" (UID: \"dbcdd6679a6e77f8381bc386784a1377\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.719812 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.719792    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:31.820229 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.820153    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:31.904351 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.904322    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.909412 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:31.909395    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:31.920920 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:31.920895    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.021458 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:32.021423    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.121944 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:32.121879    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.222420 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:32.222388    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.252518 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.252496    2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 19:38:32.252953 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.252664    2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 19:38:32.323583 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:32.323560    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.345023 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.344995    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 19:33:31 +0000 UTC" deadline="2028-04-02 21:05:03.509952903 +0000 UTC"
Sep 18 19:38:32.345123 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.345022    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13489h26m31.164933548s"
Sep 18 19:38:32.424685 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:32.424623    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.494001 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:32.493972    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80dc006f68abf1cd999156c6067d05e.slice/crio-7b4ddd3d69d8516688b5b5d7e62a40ab60814bd0fec3886c421305ab42a4f55f WatchSource:0}: Error finding container 7b4ddd3d69d8516688b5b5d7e62a40ab60814bd0fec3886c421305ab42a4f55f: Status 404 returned error can't find the container with id 7b4ddd3d69d8516688b5b5d7e62a40ab60814bd0fec3886c421305ab42a4f55f
Sep 18 19:38:32.494335 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:32.494314    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbcdd6679a6e77f8381bc386784a1377.slice/crio-4d9c6094971c0b3f5c3a5be1762adccc346f28cf9f3508c86e59dfe8a28f8568 WatchSource:0}: Error finding container 4d9c6094971c0b3f5c3a5be1762adccc346f28cf9f3508c86e59dfe8a28f8568: Status 404 returned error can't find the container with id 4d9c6094971c0b3f5c3a5be1762adccc346f28cf9f3508c86e59dfe8a28f8568
Sep 18 19:38:32.499264 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.499248    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:38:32.525681 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:32.525648    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.626099 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:32.626067    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.678322 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.678258    2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:32.727141 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:32.727115    2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-190.ec2.internal\" not found"
Sep 18 19:38:32.757630 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.757610    2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:32.760122 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.760107    2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:32.815798 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.815779    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:32.829304 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.829288    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:32.830939 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.830907    2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal"
Sep 18 19:38:32.839116 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:32.839101    2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 19:38:33.135552 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.135367    2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 19:38:33.296791 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.296753    2559 apiserver.go:52] "Watching apiserver"
Sep 18 19:38:33.301936 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.301915    2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 19:38:33.302354 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.302326    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-hvl8z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal","openshift-multus/network-metrics-daemon-vfx7n","openshift-network-diagnostics/network-check-target-86rp7","openshift-network-operator/iptables-alerter-jhjbw","openshift-ovn-kubernetes/ovnkube-node-dt7qz","kube-system/global-pull-secret-syncer-26877","kube-system/konnectivity-agent-5bdnh","kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hdqj7","openshift-image-registry/node-ca-ccfk8","openshift-multus/multus-additional-cni-plugins-89bjd","openshift-multus/multus-g6zjk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"]
Sep 18 19:38:33.305046 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.305025    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.305144 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.305103    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:33.307310 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.307250    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.310025 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.309392    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 19:38:33.310025 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.309678    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 19:38:33.310025 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.309885    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ttxcp\""
Sep 18 19:38:33.310386 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.310364    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 19:38:33.310630 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.310589    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 19:38:33.310718 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.310686    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:33.310794 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.310761    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:33.311653 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.311633    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:33.311741 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.311698    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:33.313811 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.313772    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.315658 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.315641    2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 19:38:33.315923 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.315759    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:33.315923 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.315786    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 19:38:33.315923 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.315812    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:33.316130 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.316110    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-59szm\""
Sep 18 19:38:33.316224 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.316206    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.318214 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.318195    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 19:38:33.318314 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.318298    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 19:38:33.318384 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.318350    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 19:38:33.318448 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.318431    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 19:38:33.318502 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.318478    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 19:38:33.318940 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.318673    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q9qnn\""
Sep 18 19:38:33.318940 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.318702    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.318940 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.318712    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 19:38:33.321617 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.321159    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 19:38:33.321617 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.321418    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dv5gn\""
Sep 18 19:38:33.321617 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.321458    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 19:38:33.321617 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.321564    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 19:38:33.324161 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.324049    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.325876 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.325859    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 19:38:33.326175 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326158    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 19:38:33.326376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-kubelet\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.326487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326386    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-etc-kubernetes\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.326487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326412    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhbc6\" (UniqueName: \"kubernetes.io/projected/d271732b-e2a8-4cc5-b5e6-6481da077fbb-kube-api-access-qhbc6\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:33.326487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326421    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sndxv\""
Sep 18 19:38:33.326487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326436    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.326487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326464    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovnkube-config\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.326487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326486    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-env-overrides\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326511    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/324f320a-8733-4798-a3a8-9541f5119139-cni-binary-copy\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326534    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-cni-bin\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-ovn\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326586    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-node-log\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326610    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-log-socket\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326643    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-cni-bin\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326668    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-k8s-cni-cncf-io\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326693    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/324f320a-8733-4798-a3a8-9541f5119139-multus-daemon-config\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326708    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/595bb485-5ce7-4268-97b9-df107f423e28-iptables-alerter-script\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-var-lib-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.326771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326772    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65111b60-974a-4cc8-8faa-73e99a98233a-kubelet-config\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326841    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-system-cni-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326866    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-socket-dir-parent\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326889    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326911    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-systemd\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326945    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-etc-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.326999    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65111b60-974a-4cc8-8faa-73e99a98233a-dbus\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327037    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327049    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-cni-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327072    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-slash\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327097    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327121    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovnkube-script-lib\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327149    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmgw\" (UniqueName: \"kubernetes.io/projected/f0aed8e6-4720-49e3-96e7-2826644f38b7-kube-api-access-jkmgw\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327171    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-cni-multus\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327193    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/595bb485-5ce7-4268-97b9-df107f423e28-host-slash\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.327263 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327215    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnhr5\" (UniqueName: \"kubernetes.io/projected/595bb485-5ce7-4268-97b9-df107f423e28-kube-api-access-dnhr5\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327235    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-kubelet\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327259    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-systemd-units\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327298    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-run-netns\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327321    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-cni-netd\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327343    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovn-node-metrics-cert\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327374    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-cnibin\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327394    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-netns\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327406    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-hostroot\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327418    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-conf-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327439    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvdh\" (UniqueName: \"kubernetes.io/projected/324f320a-8733-4798-a3a8-9541f5119139-kube-api-access-fwvdh\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327479    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:33.328180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.327525    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-multus-certs\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.329252 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.329081    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 19:38:33.329252 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.329106    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 19:38:33.329252 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.329104    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xr62l\""
Sep 18 19:38:33.329424 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.329408    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.330003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.329987    2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 19:38:33.331327 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.331166    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 19:38:33.331327 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.331224    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hvxzv\""
Sep 18 19:38:33.331465 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.331407    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 19:38:33.331805 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.331790    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.333797 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.333778    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tgsvg\""
Sep 18 19:38:33.333888 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.333813    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 19:38:33.333888 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.333797    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 19:38:33.333888 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.333853    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 19:38:33.334460 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.334443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.336400 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.336382    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-v7gsv\""
Sep 18 19:38:33.336400 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.336398    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 19:38:33.336539 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.336418    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 19:38:33.362696 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.362674    2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n4vmw"
Sep 18 19:38:33.371316 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.371294    2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n4vmw"
Sep 18 19:38:33.416576 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.416511    2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 19:38:33.428273 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-cni-bin\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.428389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-log-socket\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.428389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428297    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-cni-bin\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.428389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428321    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/324f320a-8733-4798-a3a8-9541f5119139-multus-daemon-config\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.428389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428366    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-cni-bin\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.428389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428377    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-log-socket\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428428    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-cni-bin\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428448    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65111b60-974a-4cc8-8faa-73e99a98233a-kubelet-config\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428474    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-system-cni-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428492    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-socket-dir-parent\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428515    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428537    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428559    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmgw\" (UniqueName: \"kubernetes.io/projected/f0aed8e6-4720-49e3-96e7-2826644f38b7-kube-api-access-jkmgw\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428585    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-registration-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428609    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-system-cni-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.428631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428608    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-device-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428653    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65111b60-974a-4cc8-8faa-73e99a98233a-dbus\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-cni-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428704    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428726    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovnkube-script-lib\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428726    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65111b60-974a-4cc8-8faa-73e99a98233a-kubelet-config\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428844    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcp22\" (UniqueName: \"kubernetes.io/projected/9e8630b6-9465-4b75-aa0f-4fabd10d071a-kube-api-access-kcp22\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428856    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-socket-dir-parent\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65111b60-974a-4cc8-8faa-73e99a98233a-dbus\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428876    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-cni-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428880    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/324f320a-8733-4798-a3a8-9541f5119139-multus-daemon-config\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428904    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428878    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/000387d9-d879-418d-a2ca-a80f5883f7a5-agent-certs\") pod \"konnectivity-agent-5bdnh\" (UID: \"000387d9-d879-418d-a2ca-a80f5883f7a5\") " pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.428985    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-var-lib-kubelet\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429015    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnhr5\" (UniqueName: \"kubernetes.io/projected/595bb485-5ce7-4268-97b9-df107f423e28-kube-api-access-dnhr5\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429041    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-cni-netd\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.429144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429066    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovn-node-metrics-cert\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429185    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-cni-netd\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429196    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-socket-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429331    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysctl-d\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovnkube-script-lib\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429363    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-tuned\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429387    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-cni-binary-copy\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429414    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-cnibin\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429460    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429484    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-multus-certs\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429486    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-cnibin\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429506    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-hosts-file\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429511    2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429530    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysctl-conf\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.429545    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429567    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-kubelet\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429577    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-multus-certs\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.429937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429593    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-etc-kubernetes\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.429619    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.9296001 +0000 UTC m=+4.892332834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429633    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-kubelet\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429637    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-etc-kubernetes\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovnkube-config\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429705    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-kubernetes\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-system-cni-dir\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/324f320a-8733-4798-a3a8-9541f5119139-cni-binary-copy\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429804    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-ovn\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-node-log\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429870    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsmbr\" (UniqueName: \"kubernetes.io/projected/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-kube-api-access-bsmbr\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429895    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-tmp\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429897    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-ovn\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429922    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-k8s-cni-cncf-io\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/595bb485-5ce7-4268-97b9-df107f423e28-iptables-alerter-script\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.430647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-var-lib-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.429992    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-node-log\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430011    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysconfig\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430042    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-sys\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430039    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-k8s-cni-cncf-io\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430069    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430093    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-systemd\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430098    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-var-lib-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-etc-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430140    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-cnibin\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430164    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-slash\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430194    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovnkube-config\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430187    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hqh\" (UniqueName: \"kubernetes.io/projected/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-kube-api-access-z4hqh\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-cni-multus\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430229    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/324f320a-8733-4798-a3a8-9541f5119139-cni-binary-copy\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/595bb485-5ce7-4268-97b9-df107f423e28-host-slash\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-etc-openvswitch\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.431405 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430275    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-kubelet\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430300    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-var-lib-cni-multus\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430298    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-systemd-units\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-run-netns\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-systemd-units\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430344    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-sys-fs\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.430349    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.430401    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret podName:65111b60-974a-4cc8-8faa-73e99a98233a nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.930386313 +0000 UTC m=+4.893119031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret") pod "global-pull-secret-syncer-26877" (UID: "65111b60-974a-4cc8-8faa-73e99a98233a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430416    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-serviceca\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430435    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-kubelet\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430442    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-netns\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430414    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-run-netns\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430455    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-slash\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430459    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/595bb485-5ce7-4268-97b9-df107f423e28-host-slash\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430491    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-run-systemd\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430513    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-hostroot\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430538    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-conf-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430567    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-hostroot\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.432268 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-host-run-netns\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430614    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/324f320a-8733-4798-a3a8-9541f5119139-multus-conf-dir\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430643    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvdh\" (UniqueName: \"kubernetes.io/projected/324f320a-8733-4798-a3a8-9541f5119139-kube-api-access-fwvdh\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430669    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bggqp\" (UniqueName: \"kubernetes.io/projected/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-kube-api-access-bggqp\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-systemd\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430727    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-host\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430752    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430789    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-tmp-dir\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430811    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-run\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430853    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-lib-modules\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhbc6\" (UniqueName: \"kubernetes.io/projected/d271732b-e2a8-4cc5-b5e6-6481da077fbb-kube-api-access-qhbc6\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430906    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430925    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-env-overrides\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430946    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/000387d9-d879-418d-a2ca-a80f5883f7a5-konnectivity-ca\") pod \"konnectivity-agent-5bdnh\" (UID: \"000387d9-d879-418d-a2ca-a80f5883f7a5\") " pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430954    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f0aed8e6-4720-49e3-96e7-2826644f38b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.430976    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-host\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.431022    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-modprobe-d\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.433127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.431057    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw9z7\" (UniqueName: \"kubernetes.io/projected/70099e14-8e28-4a33-b5e2-a29a25a1e226-kube-api-access-qw9z7\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.433953 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.431374    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f0aed8e6-4720-49e3-96e7-2826644f38b7-env-overrides\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.433953 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.431506    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/595bb485-5ce7-4268-97b9-df107f423e28-iptables-alerter-script\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.434563 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.434541    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f0aed8e6-4720-49e3-96e7-2826644f38b7-ovn-node-metrics-cert\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.435731 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.435704    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:33.435844 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.435739    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:33.435844 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.435752    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7t2rb for pod openshift-network-diagnostics/network-check-target-86rp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:33.435844 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.435817    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb podName:f744fa00-c636-4c25-a403-0635a8f15523 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:33.935801941 +0000 UTC m=+4.898534660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7t2rb" (UniqueName: "kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb") pod "network-check-target-86rp7" (UID: "f744fa00-c636-4c25-a403-0635a8f15523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:33.437878 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.437854    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmgw\" (UniqueName: \"kubernetes.io/projected/f0aed8e6-4720-49e3-96e7-2826644f38b7-kube-api-access-jkmgw\") pod \"ovnkube-node-dt7qz\" (UID: \"f0aed8e6-4720-49e3-96e7-2826644f38b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.438362 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.438339    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnhr5\" (UniqueName: \"kubernetes.io/projected/595bb485-5ce7-4268-97b9-df107f423e28-kube-api-access-dnhr5\") pod \"iptables-alerter-jhjbw\" (UID: \"595bb485-5ce7-4268-97b9-df107f423e28\") " pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.438493 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.438473    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhbc6\" (UniqueName: \"kubernetes.io/projected/d271732b-e2a8-4cc5-b5e6-6481da077fbb-kube-api-access-qhbc6\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:33.439124 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.439103    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvdh\" (UniqueName: \"kubernetes.io/projected/324f320a-8733-4798-a3a8-9541f5119139-kube-api-access-fwvdh\") pod \"multus-g6zjk\" (UID: \"324f320a-8733-4798-a3a8-9541f5119139\") " pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.470791 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.470746    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal" event={"ID":"c80dc006f68abf1cd999156c6067d05e","Type":"ContainerStarted","Data":"7b4ddd3d69d8516688b5b5d7e62a40ab60814bd0fec3886c421305ab42a4f55f"}
Sep 18 19:38:33.472008 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.471985    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal" event={"ID":"dbcdd6679a6e77f8381bc386784a1377","Type":"ContainerStarted","Data":"4d9c6094971c0b3f5c3a5be1762adccc346f28cf9f3508c86e59dfe8a28f8568"}
Sep 18 19:38:33.532272 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bggqp\" (UniqueName: \"kubernetes.io/projected/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-kube-api-access-bggqp\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.532403 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532282    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-systemd\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532403 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532304    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-host\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532403 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532327    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.532403 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532353    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-tmp-dir\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.532403 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532374    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-run\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-systemd\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-lib-modules\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532437    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/000387d9-d879-418d-a2ca-a80f5883f7a5-konnectivity-ca\") pod \"konnectivity-agent-5bdnh\" (UID: \"000387d9-d879-418d-a2ca-a80f5883f7a5\") " pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532451    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-host\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532460    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-run\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532467    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-modprobe-d\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw9z7\" (UniqueName: \"kubernetes.io/projected/70099e14-8e28-4a33-b5e2-a29a25a1e226-kube-api-access-qw9z7\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532539    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532580    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-modprobe-d\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532601    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-registration-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-host\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532617    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-lib-modules\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.532646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532395    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-host\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532671    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-registration-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532712    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-tmp-dir\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532734    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-device-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532979    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.532988    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-device-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533020    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533054    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533088    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcp22\" (UniqueName: \"kubernetes.io/projected/9e8630b6-9465-4b75-aa0f-4fabd10d071a-kube-api-access-kcp22\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533119    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/000387d9-d879-418d-a2ca-a80f5883f7a5-agent-certs\") pod \"konnectivity-agent-5bdnh\" (UID: \"000387d9-d879-418d-a2ca-a80f5883f7a5\") " pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533145    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533150    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-var-lib-kubelet\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-socket-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-var-lib-kubelet\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533218    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysctl-d\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533241    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-tuned\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533266    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-cni-binary-copy\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533290    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-socket-dir\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533386    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysctl-d\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533393    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-hosts-file\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533430    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysctl-conf\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533445    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-hosts-file\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-kubernetes\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533512    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-system-cni-dir\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533552    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsmbr\" (UniqueName: \"kubernetes.io/projected/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-kube-api-access-bsmbr\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533563    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-etc-selinux\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-tmp\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533584    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysctl-conf\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533599    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysconfig\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-system-cni-dir\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.533912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533621    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-sys\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-cnibin\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hqh\" (UniqueName: \"kubernetes.io/projected/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-kube-api-access-z4hqh\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533693    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-sysconfig\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/000387d9-d879-418d-a2ca-a80f5883f7a5-konnectivity-ca\") pod \"konnectivity-agent-5bdnh\" (UID: \"000387d9-d879-418d-a2ca-a80f5883f7a5\") " pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533705    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70099e14-8e28-4a33-b5e2-a29a25a1e226-cni-binary-copy\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-kubernetes\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533707    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-sys-fs\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533727    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-cnibin\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533746    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-serviceca\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533781    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-sys\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533849    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e8630b6-9465-4b75-aa0f-4fabd10d071a-sys-fs\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.533887    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70099e14-8e28-4a33-b5e2-a29a25a1e226-tuning-conf-dir\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.534623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.534126    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-serviceca\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.535822 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.535792    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/000387d9-d879-418d-a2ca-a80f5883f7a5-agent-certs\") pod \"konnectivity-agent-5bdnh\" (UID: \"000387d9-d879-418d-a2ca-a80f5883f7a5\") " pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:33.535956 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.535938    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-tmp\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.536209 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.536192    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-etc-tuned\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.541240 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.541222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hqh\" (UniqueName: \"kubernetes.io/projected/3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f-kube-api-access-z4hqh\") pod \"tuned-hdqj7\" (UID: \"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f\") " pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.541356 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.541336    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bggqp\" (UniqueName: \"kubernetes.io/projected/d8ab3246-2a81-4ed1-8f73-2e44beaeba85-kube-api-access-bggqp\") pod \"node-resolver-hvl8z\" (UID: \"d8ab3246-2a81-4ed1-8f73-2e44beaeba85\") " pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.541615 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.541593    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsmbr\" (UniqueName: \"kubernetes.io/projected/0be5902d-0c5c-4c36-b955-a0d97e5e61f5-kube-api-access-bsmbr\") pod \"node-ca-ccfk8\" (UID: \"0be5902d-0c5c-4c36-b955-a0d97e5e61f5\") " pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.541690 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.541674    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcp22\" (UniqueName: \"kubernetes.io/projected/9e8630b6-9465-4b75-aa0f-4fabd10d071a-kube-api-access-kcp22\") pod \"aws-ebs-csi-driver-node-bksqj\" (UID: \"9e8630b6-9465-4b75-aa0f-4fabd10d071a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.542127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.542108    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw9z7\" (UniqueName: \"kubernetes.io/projected/70099e14-8e28-4a33-b5e2-a29a25a1e226-kube-api-access-qw9z7\") pod \"multus-additional-cni-plugins-89bjd\" (UID: \"70099e14-8e28-4a33-b5e2-a29a25a1e226\") " pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.624520 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.624479    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g6zjk"
Sep 18 19:38:33.631671 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.631647    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhjbw"
Sep 18 19:38:33.640372 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.640350    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:33.645036 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.645018    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj"
Sep 18 19:38:33.652523 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.652503    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hvl8z"
Sep 18 19:38:33.659061 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.659042    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:33.665657 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.665637    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hdqj7"
Sep 18 19:38:33.671095 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.671076    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ccfk8"
Sep 18 19:38:33.676694 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.676674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-89bjd"
Sep 18 19:38:33.935808 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.935741    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:33.935808 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:33.935782    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:33.935987 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.935871    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:33.935987 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.935872    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:33.935987 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.935921    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret podName:65111b60-974a-4cc8-8faa-73e99a98233a nodeName:}" failed. No retries permitted until 2026-09-18 19:38:34.935908877 +0000 UTC m=+5.898641592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret") pod "global-pull-secret-syncer-26877" (UID: "65111b60-974a-4cc8-8faa-73e99a98233a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:33.935987 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:33.935935    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:38:34.935928578 +0000 UTC m=+5.898661293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:34.036319 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.036287    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:34.036501 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.036436    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:34.036501 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.036456    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:34.036501 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.036468    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7t2rb for pod openshift-network-diagnostics/network-check-target-86rp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:34.036663 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.036537    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb podName:f744fa00-c636-4c25-a403-0635a8f15523 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:35.036518006 +0000 UTC m=+5.999250745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7t2rb" (UniqueName: "kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb") pod "network-check-target-86rp7" (UID: "f744fa00-c636-4c25-a403-0635a8f15523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:34.282227 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.282198    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70099e14_8e28_4a33_b5e2_a29a25a1e226.slice/crio-01b3c03bed3b1511d705a0905b8ff439a3176b0b296ea6b3ed49ad5c3182b862 WatchSource:0}: Error finding container 01b3c03bed3b1511d705a0905b8ff439a3176b0b296ea6b3ed49ad5c3182b862: Status 404 returned error can't find the container with id 01b3c03bed3b1511d705a0905b8ff439a3176b0b296ea6b3ed49ad5c3182b862
Sep 18 19:38:34.283808 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.283753    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324f320a_8733_4798_a3a8_9541f5119139.slice/crio-64c6e71990a1b47a79da54d14fa2e4afa819fb58732a27199c391a173e93cf63 WatchSource:0}: Error finding container 64c6e71990a1b47a79da54d14fa2e4afa819fb58732a27199c391a173e93cf63: Status 404 returned error can't find the container with id 64c6e71990a1b47a79da54d14fa2e4afa819fb58732a27199c391a173e93cf63
Sep 18 19:38:34.284935 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.284912    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ab3246_2a81_4ed1_8f73_2e44beaeba85.slice/crio-a864310341c6973d2d8e7b891371a6667f1e6d102e2f4531d86ff3eee0d14e21 WatchSource:0}: Error finding container a864310341c6973d2d8e7b891371a6667f1e6d102e2f4531d86ff3eee0d14e21: Status 404 returned error can't find the container with id a864310341c6973d2d8e7b891371a6667f1e6d102e2f4531d86ff3eee0d14e21
Sep 18 19:38:34.289266 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.289243    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0aed8e6_4720_49e3_96e7_2826644f38b7.slice/crio-4f8dbf2b0f2553b22a7f94a8494e9f3f379165dc9855d02e1258c6783494a582 WatchSource:0}: Error finding container 4f8dbf2b0f2553b22a7f94a8494e9f3f379165dc9855d02e1258c6783494a582: Status 404 returned error can't find the container with id 4f8dbf2b0f2553b22a7f94a8494e9f3f379165dc9855d02e1258c6783494a582
Sep 18 19:38:34.291391 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.291357    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000387d9_d879_418d_a2ca_a80f5883f7a5.slice/crio-1eb8dc233b23515ff5d0be180bf1e38bd8053dc58d19384bfeffe606f915a1a5 WatchSource:0}: Error finding container 1eb8dc233b23515ff5d0be180bf1e38bd8053dc58d19384bfeffe606f915a1a5: Status 404 returned error can't find the container with id 1eb8dc233b23515ff5d0be180bf1e38bd8053dc58d19384bfeffe606f915a1a5
Sep 18 19:38:34.313177 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.313152    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8630b6_9465_4b75_aa0f_4fabd10d071a.slice/crio-fefaf8626977c702fc74110b239ed700e45cfb35ee12e26ce6bee5522128b562 WatchSource:0}: Error finding container fefaf8626977c702fc74110b239ed700e45cfb35ee12e26ce6bee5522128b562: Status 404 returned error can't find the container with id fefaf8626977c702fc74110b239ed700e45cfb35ee12e26ce6bee5522128b562
Sep 18 19:38:34.314175 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.314150    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca9d706_4a2b_4f4b_8eba_4b84e2f48b5f.slice/crio-113e8b60d589f24f6306d5f141b90d0624ad928b6bb5e67ef09587cbbc41c5ce WatchSource:0}: Error finding container 113e8b60d589f24f6306d5f141b90d0624ad928b6bb5e67ef09587cbbc41c5ce: Status 404 returned error can't find the container with id 113e8b60d589f24f6306d5f141b90d0624ad928b6bb5e67ef09587cbbc41c5ce
Sep 18 19:38:34.314804 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.314787    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595bb485_5ce7_4268_97b9_df107f423e28.slice/crio-02789b4bfcf3e89a186a0ebec344e0800b8a3232fa985c7ebe9a14df554ddb1f WatchSource:0}: Error finding container 02789b4bfcf3e89a186a0ebec344e0800b8a3232fa985c7ebe9a14df554ddb1f: Status 404 returned error can't find the container with id 02789b4bfcf3e89a186a0ebec344e0800b8a3232fa985c7ebe9a14df554ddb1f
Sep 18 19:38:34.316254 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:38:34.316228    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be5902d_0c5c_4c36_b955_a0d97e5e61f5.slice/crio-d4c050441a07b0c5a8ce5c38fe124afd407dbad04c056592735d76873afdbfe0 WatchSource:0}: Error finding container d4c050441a07b0c5a8ce5c38fe124afd407dbad04c056592735d76873afdbfe0: Status 404 returned error can't find the container with id d4c050441a07b0c5a8ce5c38fe124afd407dbad04c056592735d76873afdbfe0
Sep 18 19:38:34.374547 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.374501    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:33 +0000 UTC" deadline="2028-06-05 16:57:34.877864942 +0000 UTC"
Sep 18 19:38:34.374666 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.374549    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15021h19m0.503319962s"
Sep 18 19:38:34.465213 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.465182    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:34.465369 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.465320    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:34.476179 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.475996    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5bdnh" event={"ID":"000387d9-d879-418d-a2ca-a80f5883f7a5","Type":"ContainerStarted","Data":"1eb8dc233b23515ff5d0be180bf1e38bd8053dc58d19384bfeffe606f915a1a5"}
Sep 18 19:38:34.477308 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.477282    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccfk8" event={"ID":"0be5902d-0c5c-4c36-b955-a0d97e5e61f5","Type":"ContainerStarted","Data":"d4c050441a07b0c5a8ce5c38fe124afd407dbad04c056592735d76873afdbfe0"}
Sep 18 19:38:34.478445 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.478424    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj" event={"ID":"9e8630b6-9465-4b75-aa0f-4fabd10d071a","Type":"ContainerStarted","Data":"fefaf8626977c702fc74110b239ed700e45cfb35ee12e26ce6bee5522128b562"}
Sep 18 19:38:34.480741 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.480716    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hdqj7" event={"ID":"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f","Type":"ContainerStarted","Data":"113e8b60d589f24f6306d5f141b90d0624ad928b6bb5e67ef09587cbbc41c5ce"}
Sep 18 19:38:34.482727 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.482695    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hvl8z" event={"ID":"d8ab3246-2a81-4ed1-8f73-2e44beaeba85","Type":"ContainerStarted","Data":"a864310341c6973d2d8e7b891371a6667f1e6d102e2f4531d86ff3eee0d14e21"}
Sep 18 19:38:34.483904 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.483886    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g6zjk" event={"ID":"324f320a-8733-4798-a3a8-9541f5119139","Type":"ContainerStarted","Data":"64c6e71990a1b47a79da54d14fa2e4afa819fb58732a27199c391a173e93cf63"}
Sep 18 19:38:34.486308 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.486290    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerStarted","Data":"01b3c03bed3b1511d705a0905b8ff439a3176b0b296ea6b3ed49ad5c3182b862"}
Sep 18 19:38:34.488221 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.488200    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal" event={"ID":"c80dc006f68abf1cd999156c6067d05e","Type":"ContainerStarted","Data":"52031f5fdc6144e5059822f46644a924ada849f7a9c847565f8db3e5c28b4a79"}
Sep 18 19:38:34.489684 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.489657    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhjbw" event={"ID":"595bb485-5ce7-4268-97b9-df107f423e28","Type":"ContainerStarted","Data":"02789b4bfcf3e89a186a0ebec344e0800b8a3232fa985c7ebe9a14df554ddb1f"}
Sep 18 19:38:34.491710 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.491690    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"4f8dbf2b0f2553b22a7f94a8494e9f3f379165dc9855d02e1258c6783494a582"}
Sep 18 19:38:34.507027 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.506990    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-190.ec2.internal" podStartSLOduration=2.506978815 podStartE2EDuration="2.506978815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:34.506753817 +0000 UTC m=+5.469486554" watchObservedRunningTime="2026-09-18 19:38:34.506978815 +0000 UTC m=+5.469711545"
Sep 18 19:38:34.943401 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.943283    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:34.943401 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:34.943342    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:34.943624 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.943464    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:34.943624 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.943523    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret podName:65111b60-974a-4cc8-8faa-73e99a98233a nodeName:}" failed. No retries permitted until 2026-09-18 19:38:36.943506454 +0000 UTC m=+7.906239185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret") pod "global-pull-secret-syncer-26877" (UID: "65111b60-974a-4cc8-8faa-73e99a98233a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:34.943983 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.943911    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:34.943983 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:34.943980    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:38:36.943947951 +0000 UTC m=+7.906680670 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:35.044267 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:35.044227    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:35.044470 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:35.044454    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:35.044544 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:35.044483    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:35.044544 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:35.044495    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7t2rb for pod openshift-network-diagnostics/network-check-target-86rp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:35.044646 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:35.044554    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb podName:f744fa00-c636-4c25-a403-0635a8f15523 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:37.044536519 +0000 UTC m=+8.007269250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7t2rb" (UniqueName: "kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb") pod "network-check-target-86rp7" (UID: "f744fa00-c636-4c25-a403-0635a8f15523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:35.375140 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:35.375059    2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 19:33:33 +0000 UTC" deadline="2028-04-03 09:01:18.334255233 +0000 UTC"
Sep 18 19:38:35.375140 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:35.375105    2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13501h22m42.959154697s"
Sep 18 19:38:35.468412 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:35.467715    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:35.468412 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:35.467847    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:35.468412 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:35.468254    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:35.468412 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:35.468347    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:35.506398 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:35.506359    2559 generic.go:356] "Generic (PLEG): container finished" podID="dbcdd6679a6e77f8381bc386784a1377" containerID="0b735c4696f9c124508bcfb4eacd5061d136f3c500d447b49318d0b6765f37b6" exitCode=0
Sep 18 19:38:35.506938 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:35.506913    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal" event={"ID":"dbcdd6679a6e77f8381bc386784a1377","Type":"ContainerDied","Data":"0b735c4696f9c124508bcfb4eacd5061d136f3c500d447b49318d0b6765f37b6"}
Sep 18 19:38:36.465256 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:36.465222    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:36.465732 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:36.465365    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:36.525363 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:36.524655    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal" event={"ID":"dbcdd6679a6e77f8381bc386784a1377","Type":"ContainerStarted","Data":"d0de52600b9d8e014e279494a2baa875fe78a4c8bf0054b82d587c082a10a995"}
Sep 18 19:38:36.959166 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:36.958371    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:36.959166 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:36.958434    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:36.959166 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:36.958571    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:36.959166 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:36.958636    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret podName:65111b60-974a-4cc8-8faa-73e99a98233a nodeName:}" failed. No retries permitted until 2026-09-18 19:38:40.958617387 +0000 UTC m=+11.921350110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret") pod "global-pull-secret-syncer-26877" (UID: "65111b60-974a-4cc8-8faa-73e99a98233a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:36.959166 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:36.959034    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:36.959166 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:36.959084    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:38:40.959069004 +0000 UTC m=+11.921801725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:37.061098 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:37.060507    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:37.061098 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:37.060671    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:37.061098 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:37.060693    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:37.061098 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:37.060706    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7t2rb for pod openshift-network-diagnostics/network-check-target-86rp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:37.061098 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:37.060769    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb podName:f744fa00-c636-4c25-a403-0635a8f15523 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:41.0607507 +0000 UTC m=+12.023483419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7t2rb" (UniqueName: "kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb") pod "network-check-target-86rp7" (UID: "f744fa00-c636-4c25-a403-0635a8f15523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:37.465364 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:37.465327    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:37.465851 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:37.465327    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:37.465851 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:37.465449    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:37.465851 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:37.465543    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:38.464680 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:38.464645    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:38.464834 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:38.464801    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:39.465883 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:39.465852    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:39.466373 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:39.465986    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:39.466373 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:39.466009    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:39.466373 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:39.466073    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:40.465470 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:40.465434    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:40.465651 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:40.465566    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:40.996179 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:40.996152    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:40.996601 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:40.996192    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:40.996601 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:40.996313    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:40.996601 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:40.996364    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret podName:65111b60-974a-4cc8-8faa-73e99a98233a nodeName:}" failed. No retries permitted until 2026-09-18 19:38:48.996351322 +0000 UTC m=+19.959084052 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret") pod "global-pull-secret-syncer-26877" (UID: "65111b60-974a-4cc8-8faa-73e99a98233a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:40.996601 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:40.996313    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:40.996601 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:40.996436    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:38:48.996424026 +0000 UTC m=+19.959156755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:41.097243 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:41.097207    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:41.097419 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:41.097338    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:41.097419 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:41.097362    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:41.097419 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:41.097374    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7t2rb for pod openshift-network-diagnostics/network-check-target-86rp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:41.097419 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:41.097416    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb podName:f744fa00-c636-4c25-a403-0635a8f15523 nodeName:}" failed. No retries permitted until 2026-09-18 19:38:49.097404412 +0000 UTC m=+20.060137133 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7t2rb" (UniqueName: "kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb") pod "network-check-target-86rp7" (UID: "f744fa00-c636-4c25-a403-0635a8f15523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:41.467511 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:41.467037    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:41.467511 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:41.467214    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:41.467511 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:41.467305    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:41.467511 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:41.467409    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:42.465135 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:42.465096    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:42.465565 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:42.465253    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:43.464710 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:43.464677    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:43.464890 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:43.464788    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:43.464890 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:43.464853    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:43.465007 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:43.464985    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:44.464839 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:44.464801    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:44.465240 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:44.464928    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:45.465326 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:45.465292    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:45.465778 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:45.465408    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:45.465778 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:45.465469    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:45.465778 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:45.465586    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:46.465036 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:46.465004    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:46.465238 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:46.465150    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:47.465269 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:47.465234    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:47.465647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:47.465234    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:47.465647 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:47.465347    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:47.465647 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:47.465457    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:48.465616 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:48.465577    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:48.466072 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:48.465734    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:49.053049 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:49.053012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:49.053049 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:49.053059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:49.053284 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.053164    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:49.053284 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.053223    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret podName:65111b60-974a-4cc8-8faa-73e99a98233a nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.053206264 +0000 UTC m=+36.015938984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret") pod "global-pull-secret-syncer-26877" (UID: "65111b60-974a-4cc8-8faa-73e99a98233a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:38:49.053284 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.053164    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:49.053284 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.053275    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.05326699 +0000 UTC m=+36.015999705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:38:49.154276 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:49.154234    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:49.154447 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.154417    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:38:49.154447 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.154443    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:38:49.154570 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.154456    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7t2rb for pod openshift-network-diagnostics/network-check-target-86rp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:49.154570 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.154529    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb podName:f744fa00-c636-4c25-a403-0635a8f15523 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:05.154508177 +0000 UTC m=+36.117240893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7t2rb" (UniqueName: "kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb") pod "network-check-target-86rp7" (UID: "f744fa00-c636-4c25-a403-0635a8f15523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:38:49.465512 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:49.465474    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:49.465713 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.465606    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:49.465713 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:49.465665    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:49.466144 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:49.465786    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:50.465518 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:50.465484    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:50.465676 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:50.465612    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:51.465893 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:51.465861    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:51.466287 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:51.465940    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:51.466287 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:51.466005    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:51.466287 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:51.466087    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:52.464798 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.464619    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:52.464925 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:52.464898    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:52.558853 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.558761    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5bdnh" event={"ID":"000387d9-d879-418d-a2ca-a80f5883f7a5","Type":"ContainerStarted","Data":"690284a2c19d5237de6e7723ee21321b4307d60e437c08c4eb458e2c6764d347"}
Sep 18 19:38:52.560631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.560425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ccfk8" event={"ID":"0be5902d-0c5c-4c36-b955-a0d97e5e61f5","Type":"ContainerStarted","Data":"a57dc3063a30c9f5a2dc36bf6b8edf7cb4d6a366733e474a10840ca21b126aed"}
Sep 18 19:38:52.562106 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.562078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj" event={"ID":"9e8630b6-9465-4b75-aa0f-4fabd10d071a","Type":"ContainerStarted","Data":"be68fb598800dc95a4e6d893d6a41e9232fe391e7dc23fc32c3834943bf739ac"}
Sep 18 19:38:52.564026 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.563794    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hdqj7" event={"ID":"3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f","Type":"ContainerStarted","Data":"a446f64e2bf29a28ef2a8dace63736b0632f855398c98d7667381c4e946a35c7"}
Sep 18 19:38:52.565401 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.565372    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hvl8z" event={"ID":"d8ab3246-2a81-4ed1-8f73-2e44beaeba85","Type":"ContainerStarted","Data":"922c76b5e8a08891ce9da1f6861d7ff5a0d6b84d5533e5136db43e94b238966d"}
Sep 18 19:38:52.567093 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.567073    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g6zjk" event={"ID":"324f320a-8733-4798-a3a8-9541f5119139","Type":"ContainerStarted","Data":"fa2bfcf6b9d9fab570db302cc229e05b37cc887b8b6ff1a5c343d008e58cc0f6"}
Sep 18 19:38:52.568681 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.568658    2559 generic.go:356] "Generic (PLEG): container finished" podID="70099e14-8e28-4a33-b5e2-a29a25a1e226" containerID="98d61de6224b73b4084295e61c34161a05125860493552cac708b137422e04d5" exitCode=0
Sep 18 19:38:52.568756 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.568732    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerDied","Data":"98d61de6224b73b4084295e61c34161a05125860493552cac708b137422e04d5"}
Sep 18 19:38:52.571996 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.571976    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:38:52.572432 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.572411    2559 generic.go:356] "Generic (PLEG): container finished" podID="f0aed8e6-4720-49e3-96e7-2826644f38b7" containerID="2236d8020f1a603d5a0dcc8ed505607ecc8659c46e7b3a78b71e745afa58ab60" exitCode=1
Sep 18 19:38:52.572531 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.572443    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"c392cb1618f1faffdd433131848a976682bae66fffbaa14b291869ab031ecc17"}
Sep 18 19:38:52.572531 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.572504    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"2618bbd30bd0bbac80330f1b5653610d09ff448d2da3cd06ffb377aa3e0f7067"}
Sep 18 19:38:52.572633 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.572565    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"205822e39b8d45c2b592e9a7575902e07f479a27cda7d71593386add94cd93d8"}
Sep 18 19:38:52.572633 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.572615    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"b352109769d8d96610ddfa10578e645e4fd21bde03b0653d454b97b25d781691"}
Sep 18 19:38:52.572731 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.572661    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerDied","Data":"2236d8020f1a603d5a0dcc8ed505607ecc8659c46e7b3a78b71e745afa58ab60"}
Sep 18 19:38:52.572731 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.572712    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"545e41867b3a4126982b4c58c42d26f99ed34f61cc85c4ab4694ca922bb8963d"}
Sep 18 19:38:52.574767 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.574722    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-190.ec2.internal" podStartSLOduration=20.574710199 podStartE2EDuration="20.574710199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:36.54232473 +0000 UTC m=+7.505057469" watchObservedRunningTime="2026-09-18 19:38:52.574710199 +0000 UTC m=+23.537442937"
Sep 18 19:38:52.575180 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.575142    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5bdnh" podStartSLOduration=11.689568339000001 podStartE2EDuration="21.575130975s" totalImagesPullingTime="9.885562636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:52.574266189 +0000 UTC m=+23.536999068" watchObservedRunningTime="2026-09-18 19:38:52.575130975 +0000 UTC m=+23.537863714"
Sep 18 19:38:52.590319 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.590272    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hdqj7" podStartSLOduration=4.254327975 podStartE2EDuration="21.590258937s" totalImagesPullingTime="17.335930962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:52.590105623 +0000 UTC m=+23.552838360" watchObservedRunningTime="2026-09-18 19:38:52.590258937 +0000 UTC m=+23.552991674"
Sep 18 19:38:52.606731 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.606675    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g6zjk" podStartSLOduration=4.188748023 podStartE2EDuration="21.60666062s" totalImagesPullingTime="17.417912597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:52.606041803 +0000 UTC m=+23.568774541" watchObservedRunningTime="2026-09-18 19:38:52.60666062 +0000 UTC m=+23.569393359"
Sep 18 19:38:52.618831 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.618775    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hvl8z" podStartSLOduration=4.258528417 podStartE2EDuration="21.61875543s" totalImagesPullingTime="17.360227013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:52.618704747 +0000 UTC m=+23.581437486" watchObservedRunningTime="2026-09-18 19:38:52.61875543 +0000 UTC m=+23.581488167"
Sep 18 19:38:52.631650 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.631606    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ccfk8" podStartSLOduration=4.301198714 podStartE2EDuration="21.631591821s" totalImagesPullingTime="17.330393107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:52.631571019 +0000 UTC m=+23.594303764" watchObservedRunningTime="2026-09-18 19:38:52.631591821 +0000 UTC m=+23.594324559"
Sep 18 19:38:52.723298 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.723257    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:52.794094 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:52.794057    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:53.090564 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.090539    2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 19:38:53.410438 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.410339    2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T19:38:53.09055872Z","UUID":"805dd8b3-5b6a-4771-9de9-12c1c9ebcae9","Handler":null,"Name":"","Endpoint":""}
Sep 18 19:38:53.412236 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.412204    2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 19:38:53.412236 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.412240    2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 19:38:53.465156 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.465126    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:53.465333 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.465126    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:53.465333 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:53.465258    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:53.465333 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:53.465318    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:53.577439 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.577403    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj" event={"ID":"9e8630b6-9465-4b75-aa0f-4fabd10d071a","Type":"ContainerStarted","Data":"be46e7050753d1b19bddc6b5d3ec0653983019622f707d5592a46eec99337f9a"}
Sep 18 19:38:53.579624 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.579585    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhjbw" event={"ID":"595bb485-5ce7-4268-97b9-df107f423e28","Type":"ContainerStarted","Data":"62f5d0a0150e4e6a66f0efa0031caf69ee1aac18b33e7dd53667f776da8e6357"}
Sep 18 19:38:53.596111 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.596053    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jhjbw" podStartSLOduration=5.266029786 podStartE2EDuration="22.596023276s" totalImagesPullingTime="17.32999349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:53.59526962 +0000 UTC m=+24.558002360" watchObservedRunningTime="2026-09-18 19:38:53.596023276 +0000 UTC m=+24.558756014"
Sep 18 19:38:53.713539 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.713498    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:53.770337 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:53.770301    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:54.270125 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:54.270097    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:54.465152 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:54.465125    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:54.465283 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:54.465261    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:54.586120 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:54.586094    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:38:54.586725 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:54.586696    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"41671a5fdc95f5331a29200941375408e00892fd58268746d3380cb76faf4d13"}
Sep 18 19:38:54.589370 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:54.589340    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj" event={"ID":"9e8630b6-9465-4b75-aa0f-4fabd10d071a","Type":"ContainerStarted","Data":"dddb8f32a792a34513730fd2ec306991781956f533c0a9e4109faac5d4105b79"}
Sep 18 19:38:54.608072 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:54.608021    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bksqj" podStartSLOduration=4.073994642 podStartE2EDuration="23.6080034s" totalImagesPullingTime="19.534008758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:54.607291207 +0000 UTC m=+25.570023946" watchObservedRunningTime="2026-09-18 19:38:54.6080034 +0000 UTC m=+25.570736137"
Sep 18 19:38:55.164648 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:55.164613    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:55.165481 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:55.165458    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:55.465570 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:55.465536    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:55.465756 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:55.465537    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:55.465756 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:55.465674    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:55.465861 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:55.465829    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:55.594464 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:55.594429    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:55.595256 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:55.595237    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5bdnh"
Sep 18 19:38:55.659824 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:55.659789    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:38:56.465346 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:56.465308    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:56.465511 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:56.465466    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:57.465815 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.465643    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:57.466549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.465718    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:57.466549 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:57.465888    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:57.466549 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:57.466011    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:57.599528 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.599493    2559 generic.go:356] "Generic (PLEG): container finished" podID="70099e14-8e28-4a33-b5e2-a29a25a1e226" containerID="947f02f76864766be21533a0f2d1d1fd89fa9aab17840d392465bef3a7ee0078" exitCode=0
Sep 18 19:38:57.599689 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.599581    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerDied","Data":"947f02f76864766be21533a0f2d1d1fd89fa9aab17840d392465bef3a7ee0078"}
Sep 18 19:38:57.603139 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.603120    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:38:57.603554 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.603532    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"f475bbe4315275419974dfa66f7e623b9631ead777ba86e2416548329dca28c6"}
Sep 18 19:38:57.603839 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.603804    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:57.604107 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.603878    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:57.604107 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.603954    2559 scope.go:117] "RemoveContainer" containerID="2236d8020f1a603d5a0dcc8ed505607ecc8659c46e7b3a78b71e745afa58ab60"
Sep 18 19:38:57.619805 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:57.619786    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:58.464633 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.464604    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:58.464790 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:58.464701    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:58.608888 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.608675    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerStarted","Data":"15a009a545ba6480ff29620df94de529a31bae4da6631e6808dd72d390699927"}
Sep 18 19:38:58.612461 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.612442    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:38:58.612823 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.612804    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" event={"ID":"f0aed8e6-4720-49e3-96e7-2826644f38b7","Type":"ContainerStarted","Data":"a88d0b38c77b0f64bdb82973005da9ee2800ad3f66d80b5f56079feeb7b7d4cf"}
Sep 18 19:38:58.613106 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.613085    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:58.627637 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.627617    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:38:58.691945 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.691899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz" podStartSLOduration=10.332002612 podStartE2EDuration="27.691881991s" totalImagesPullingTime="17.359879379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:38:58.652244686 +0000 UTC m=+29.614977425" watchObservedRunningTime="2026-09-18 19:38:58.691881991 +0000 UTC m=+29.654614731"
Sep 18 19:38:58.812523 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.812423    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-26877"]
Sep 18 19:38:58.812695 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.812569    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:38:58.812695 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:58.812666    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:38:58.818505 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.818476    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-86rp7"]
Sep 18 19:38:58.818634 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.818599    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:38:58.818729 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:58.818705    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:38:58.819359 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.819336    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfx7n"]
Sep 18 19:38:58.819466 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:58.819432    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:38:58.819524 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:38:58.819509    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:38:59.618480 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:59.618446    2559 generic.go:356] "Generic (PLEG): container finished" podID="70099e14-8e28-4a33-b5e2-a29a25a1e226" containerID="15a009a545ba6480ff29620df94de529a31bae4da6631e6808dd72d390699927" exitCode=0
Sep 18 19:38:59.618992 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:38:59.618533    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerDied","Data":"15a009a545ba6480ff29620df94de529a31bae4da6631e6808dd72d390699927"}
Sep 18 19:39:00.465664 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:00.465628    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:00.465825 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:00.465673    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:39:00.465825 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:00.465733    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:39:00.465825 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:00.465752    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:39:00.465825 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:00.465817    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:39:00.465951 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:00.465911    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:39:00.621521 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:00.620952    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:00.707568 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:00.707535    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:01.625689 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:01.625648    2559 generic.go:356] "Generic (PLEG): container finished" podID="70099e14-8e28-4a33-b5e2-a29a25a1e226" containerID="f65bb709017b17b508bbdeab4c39379e22cf61c0138b6cab5b8f62d5964bf3ff" exitCode=0
Sep 18 19:39:01.626132 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:01.625707    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerDied","Data":"f65bb709017b17b508bbdeab4c39379e22cf61c0138b6cab5b8f62d5964bf3ff"}
Sep 18 19:39:01.836564 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:01.836526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:01.890717 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:01.890679    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:02.464682 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:02.464650    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:39:02.464882 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:02.464662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:02.464882 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:02.464771    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:39:02.464882 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:02.464662    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:39:02.464882 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:02.464846    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:39:02.465066 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:02.464897    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:39:03.384037 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:03.384000    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:04.465225 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:04.465188    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:04.465705 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:04.465188    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:39:04.465705 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:04.465311    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-86rp7" podUID="f744fa00-c636-4c25-a403-0635a8f15523"
Sep 18 19:39:04.465705 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:04.465406    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26877" podUID="65111b60-974a-4cc8-8faa-73e99a98233a"
Sep 18 19:39:04.465705 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:04.465188    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:39:04.465705 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:04.465519    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:39:04.653019 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:04.652981    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:04.703511 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:04.703474    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:05.073522 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.073488    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:39:05.073707 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.073575    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:39:05.073707 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:05.073641    2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:05.073707 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:05.073674    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:05.073868 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:05.073716    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret podName:65111b60-974a-4cc8-8faa-73e99a98233a nodeName:}" failed. No retries permitted until 2026-09-18 19:39:37.073693433 +0000 UTC m=+68.036426149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret") pod "global-pull-secret-syncer-26877" (UID: "65111b60-974a-4cc8-8faa-73e99a98233a") : object "kube-system"/"original-pull-secret" not registered
Sep 18 19:39:05.073868 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:05.073738    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:39:37.073728702 +0000 UTC m=+68.036461425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 19:39:05.173974 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.173921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:05.174180 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:05.174072    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 19:39:05.174180 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:05.174091    2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 19:39:05.174180 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:05.174102    2559 projected.go:196] Error preparing data for projected volume kube-api-access-7t2rb for pod openshift-network-diagnostics/network-check-target-86rp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:05.174180 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:05.174165    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb podName:f744fa00-c636-4c25-a403-0635a8f15523 nodeName:}" failed. No retries permitted until 2026-09-18 19:39:37.174149049 +0000 UTC m=+68.136881776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7t2rb" (UniqueName: "kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb") pod "network-check-target-86rp7" (UID: "f744fa00-c636-4c25-a403-0635a8f15523") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 19:39:05.879441 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.879411    2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-190.ec2.internal" event="NodeReady"
Sep 18 19:39:05.880026 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.879595    2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 19:39:05.928208 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.928176    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g792g"]
Sep 18 19:39:05.959377 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.959349    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fpsxk"]
Sep 18 19:39:05.959516 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.959481    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:05.961633 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.961612    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 19:39:05.961765 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.961612    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 19:39:05.961837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.961822    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j9gtw\""
Sep 18 19:39:05.982216 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.982193    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g792g"]
Sep 18 19:39:05.982331 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.982228    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fpsxk"]
Sep 18 19:39:05.982331 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.982303    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fpsxk"
Sep 18 19:39:05.984803 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.984785    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 19:39:05.984913 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.984802    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 19:39:05.984913 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.984816    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kvwkp\""
Sep 18 19:39:05.984913 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:05.984810    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 19:39:06.036846 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.036818    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:06.080755 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.080729    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-tmp-dir\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.080833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.080782    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.080833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.080806    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7mhj\" (UniqueName: \"kubernetes.io/projected/789fc76c-eb30-45d8-9e8f-a6ac1f6fb324-kube-api-access-m7mhj\") pod \"ingress-canary-fpsxk\" (UID: \"789fc76c-eb30-45d8-9e8f-a6ac1f6fb324\") " pod="openshift-ingress-canary/ingress-canary-fpsxk"
Sep 18 19:39:06.080833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.080830    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/789fc76c-eb30-45d8-9e8f-a6ac1f6fb324-cert\") pod \"ingress-canary-fpsxk\" (UID: \"789fc76c-eb30-45d8-9e8f-a6ac1f6fb324\") " pod="openshift-ingress-canary/ingress-canary-fpsxk"
Sep 18 19:39:06.080935 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.080847    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-config-volume\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.080935 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.080871    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4nk\" (UniqueName: \"kubernetes.io/projected/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-kube-api-access-sg4nk\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.181475 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.181450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.181604 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.181498    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7mhj\" (UniqueName: \"kubernetes.io/projected/789fc76c-eb30-45d8-9e8f-a6ac1f6fb324-kube-api-access-m7mhj\") pod \"ingress-canary-fpsxk\" (UID: \"789fc76c-eb30-45d8-9e8f-a6ac1f6fb324\") " pod="openshift-ingress-canary/ingress-canary-fpsxk"
Sep 18 19:39:06.181604 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.181541    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/789fc76c-eb30-45d8-9e8f-a6ac1f6fb324-cert\") pod \"ingress-canary-fpsxk\" (UID: \"789fc76c-eb30-45d8-9e8f-a6ac1f6fb324\") " pod="openshift-ingress-canary/ingress-canary-fpsxk"
Sep 18 19:39:06.181604 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.181563    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-config-volume\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.181833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.181613    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4nk\" (UniqueName: \"kubernetes.io/projected/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-kube-api-access-sg4nk\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.181833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.181640    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-tmp-dir\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.181833 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:06.181671    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:06.181833 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:06.181722    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:39:06.681704811 +0000 UTC m=+37.644437548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:06.182051 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.182025    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-tmp-dir\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.182349 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.182330    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-config-volume\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.185881 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.185863    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/789fc76c-eb30-45d8-9e8f-a6ac1f6fb324-cert\") pod \"ingress-canary-fpsxk\" (UID: \"789fc76c-eb30-45d8-9e8f-a6ac1f6fb324\") " pod="openshift-ingress-canary/ingress-canary-fpsxk"
Sep 18 19:39:06.189533 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.189512    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4nk\" (UniqueName: \"kubernetes.io/projected/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-kube-api-access-sg4nk\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.189796 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.189777    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7mhj\" (UniqueName: \"kubernetes.io/projected/789fc76c-eb30-45d8-9e8f-a6ac1f6fb324-kube-api-access-m7mhj\") pod \"ingress-canary-fpsxk\" (UID: \"789fc76c-eb30-45d8-9e8f-a6ac1f6fb324\") " pod="openshift-ingress-canary/ingress-canary-fpsxk"
Sep 18 19:39:06.289797 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.289773    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fpsxk"
Sep 18 19:39:06.465096 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.465060    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:39:06.465096 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.465087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:39:06.465314 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.465087    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:06.468075 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.468054    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:06.468220 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.468059    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:06.468310 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.468293    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nv7t\""
Sep 18 19:39:06.468370 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.468321    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:06.468430 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.468375    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:06.468985 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.468972    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nndrn\""
Sep 18 19:39:06.617689 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.617659    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fpsxk"]
Sep 18 19:39:06.621512 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:39:06.621486    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789fc76c_eb30_45d8_9e8f_a6ac1f6fb324.slice/crio-eea42a80482958c285ca96962c9dc606f70f76f5ac70bc86e589fde554835362 WatchSource:0}: Error finding container eea42a80482958c285ca96962c9dc606f70f76f5ac70bc86e589fde554835362: Status 404 returned error can't find the container with id eea42a80482958c285ca96962c9dc606f70f76f5ac70bc86e589fde554835362
Sep 18 19:39:06.640913 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.640889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fpsxk" event={"ID":"789fc76c-eb30-45d8-9e8f-a6ac1f6fb324","Type":"ContainerStarted","Data":"eea42a80482958c285ca96962c9dc606f70f76f5ac70bc86e589fde554835362"}
Sep 18 19:39:06.643467 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.643444    2559 generic.go:356] "Generic (PLEG): container finished" podID="70099e14-8e28-4a33-b5e2-a29a25a1e226" containerID="d329f04d411ba309379a7314ecec8a2e8d30f0af20afc630d14b5d6b08345aba" exitCode=0
Sep 18 19:39:06.643560 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.643478    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerDied","Data":"d329f04d411ba309379a7314ecec8a2e8d30f0af20afc630d14b5d6b08345aba"}
Sep 18 19:39:06.686371 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:06.686341    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:06.686485 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:06.686471    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:06.686545 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:06.686535    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:39:07.686516579 +0000 UTC m=+38.649249316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:07.650756 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:07.650422    2559 generic.go:356] "Generic (PLEG): container finished" podID="70099e14-8e28-4a33-b5e2-a29a25a1e226" containerID="f10f9663498b22033399dce12d6a8983c8a76817da1798e2b03baf1a80077f9d" exitCode=0
Sep 18 19:39:07.650756 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:07.650459    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerDied","Data":"f10f9663498b22033399dce12d6a8983c8a76817da1798e2b03baf1a80077f9d"}
Sep 18 19:39:07.696207 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:07.696179    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:07.696333 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:07.696295    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:07.696400 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:07.696356    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:39:09.696335575 +0000 UTC m=+40.659068311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:08.658700 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:08.658660    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-89bjd" event={"ID":"70099e14-8e28-4a33-b5e2-a29a25a1e226","Type":"ContainerStarted","Data":"e5a3b23b340feb3734252529536183b37d851fe5643bb512de4924c8812011f6"}
Sep 18 19:39:08.683081 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:08.683037    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-89bjd" podStartSLOduration=9.801407987 podStartE2EDuration="37.683022492s" totalImagesPullingTime="27.881614505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:08.681938865 +0000 UTC m=+39.644671603" watchObservedRunningTime="2026-09-18 19:39:08.683022492 +0000 UTC m=+39.645755231"
Sep 18 19:39:08.805546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:08.805512    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:08.866358 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:08.866320    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.662563 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:09.662522    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fpsxk" event={"ID":"789fc76c-eb30-45d8-9e8f-a6ac1f6fb324","Type":"ContainerStarted","Data":"2edbdbc873994fe6f739d00c91a45d40cb0588ddf75d6de3cae2c298f714da3b"}
Sep 18 19:39:09.679272 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:09.679222    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fpsxk" podStartSLOduration=2.275547404 podStartE2EDuration="4.679206432s" totalImagesPullingTime="2.403659028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:09.6781239 +0000 UTC m=+40.640856628" watchObservedRunningTime="2026-09-18 19:39:09.679206432 +0000 UTC m=+40.641939174"
Sep 18 19:39:09.710284 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:09.710255    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:09.710440 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:09.710422    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:09.710520 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:09.710500    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:39:13.710479007 +0000 UTC m=+44.673211742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:09.855637 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:09.855602    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:09.912707 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:09.912632    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:13.736089 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:13.736049    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:13.736477 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:13.736183    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:13.736477 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:13.736235    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:39:21.73622123 +0000 UTC m=+52.698953946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:21.790642 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:21.790529    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:21.791616 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:21.790755    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:21.796067 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:21.792011    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:39:37.791897143 +0000 UTC m=+68.754629893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:22.821599 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:22.821560    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:22.873120 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:22.873088    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:30.639481 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:30.639453    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dt7qz"
Sep 18 19:39:30.751590 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:30.751555    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:30.807678 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:30.807640    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:30.859548 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:30.859511    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:39:37.104850 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.104808    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:39:37.105352 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.104861    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:39:37.107745 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.107718    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 19:39:37.107745 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.107732    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 19:39:37.115424 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:37.115395    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:39:37.115492 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:37.115481    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:40:41.115459059 +0000 UTC m=+132.078191780 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : secret "metrics-daemon-secret" not found
Sep 18 19:39:37.118843 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.118822    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65111b60-974a-4cc8-8faa-73e99a98233a-original-pull-secret\") pod \"global-pull-secret-syncer-26877\" (UID: \"65111b60-974a-4cc8-8faa-73e99a98233a\") " pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:39:37.205357 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.205319    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:37.207733 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.207715    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 19:39:37.218295 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.218275    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 19:39:37.229312 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.229283    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2rb\" (UniqueName: \"kubernetes.io/projected/f744fa00-c636-4c25-a403-0635a8f15523-kube-api-access-7t2rb\") pod \"network-check-target-86rp7\" (UID: \"f744fa00-c636-4c25-a403-0635a8f15523\") " pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:37.380533 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.380443    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26877"
Sep 18 19:39:37.387764 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.387740    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nndrn\""
Sep 18 19:39:37.396594 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.396571    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:37.648427 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.648360    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-26877"]
Sep 18 19:39:37.651523 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:39:37.651496    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65111b60_974a_4cc8_8faa_73e99a98233a.slice/crio-9251d05aafb48bc2665ef25893096cd24d0a691eb5c1571545ba33017512279b WatchSource:0}: Error finding container 9251d05aafb48bc2665ef25893096cd24d0a691eb5c1571545ba33017512279b: Status 404 returned error can't find the container with id 9251d05aafb48bc2665ef25893096cd24d0a691eb5c1571545ba33017512279b
Sep 18 19:39:37.669606 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.669584    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-86rp7"]
Sep 18 19:39:37.672383 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:39:37.672358    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf744fa00_c636_4c25_a403_0635a8f15523.slice/crio-7c8b705b9033ae3965f7d9395981a842cfaa879db605056a22f4dd30fc738d39 WatchSource:0}: Error finding container 7c8b705b9033ae3965f7d9395981a842cfaa879db605056a22f4dd30fc738d39: Status 404 returned error can't find the container with id 7c8b705b9033ae3965f7d9395981a842cfaa879db605056a22f4dd30fc738d39
Sep 18 19:39:37.731633 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.731599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-86rp7" event={"ID":"f744fa00-c636-4c25-a403-0635a8f15523","Type":"ContainerStarted","Data":"7c8b705b9033ae3965f7d9395981a842cfaa879db605056a22f4dd30fc738d39"}
Sep 18 19:39:37.736126 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.736092    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-26877" event={"ID":"65111b60-974a-4cc8-8faa-73e99a98233a","Type":"ContainerStarted","Data":"9251d05aafb48bc2665ef25893096cd24d0a691eb5c1571545ba33017512279b"}
Sep 18 19:39:37.809896 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:37.809865    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:39:37.810044 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:37.810027    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:39:37.810111 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:39:37.810096    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:40:09.81008174 +0000 UTC m=+100.772814456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:39:42.751000 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:42.750845    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-86rp7" event={"ID":"f744fa00-c636-4c25-a403-0635a8f15523","Type":"ContainerStarted","Data":"d9eaf3749e78ad3d1e710721e37a680ec05ae8787c9c40fa75db57348719f469"}
Sep 18 19:39:42.751000 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:42.750991    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:39:42.752442 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:42.752417    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-26877" event={"ID":"65111b60-974a-4cc8-8faa-73e99a98233a","Type":"ContainerStarted","Data":"9cdb61fab50cebe10a793d89733a24a5318361273df561304072c12cd4a63b11"}
Sep 18 19:39:42.766940 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:42.766899    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-86rp7" podStartSLOduration=67.520002079 podStartE2EDuration="1m11.766888475s" totalImagesPullingTime="4.246886396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:42.765988582 +0000 UTC m=+73.728721301" watchObservedRunningTime="2026-09-18 19:39:42.766888475 +0000 UTC m=+73.729621206"
Sep 18 19:39:42.780673 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:42.780633    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-26877" podStartSLOduration=67.50751908 podStartE2EDuration="1m11.780624494s" totalImagesPullingTime="4.273105414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:42.779696311 +0000 UTC m=+73.742429049" watchObservedRunningTime="2026-09-18 19:39:42.780624494 +0000 UTC m=+73.743357231"
Sep 18 19:39:51.111374 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.111341    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"]
Sep 18 19:39:51.113215 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.113201    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"
Sep 18 19:39:51.118600 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.116651    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 19:39:51.118600 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.116936    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 19:39:51.118600 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.117184    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 19:39:51.118600 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.117396    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 19:39:51.118600 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.117606    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-xcqsb\""
Sep 18 19:39:51.123102 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.123081    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"]
Sep 18 19:39:51.143479 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.143452    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"]
Sep 18 19:39:51.145277 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.145262    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.147626 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.147608    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 18 19:39:51.147737 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.147645    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 18 19:39:51.147737 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.147650    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 18 19:39:51.147940 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.147924    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 18 19:39:51.156698 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.156678    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"]
Sep 18 19:39:51.202516 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.202490    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.202671 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.202539    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/029efa89-69ea-4a94-b144-2fac56abaab4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54ff84c98b-2dvr8\" (UID: \"029efa89-69ea-4a94-b144-2fac56abaab4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"
Sep 18 19:39:51.202671 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.202616    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-hub\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.202760 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.202692    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.202760 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.202711    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgwq4\" (UniqueName: \"kubernetes.io/projected/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-kube-api-access-lgwq4\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.202760 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.202738    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-ca\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.202760 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.202753    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.202905 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.202770    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp8cl\" (UniqueName: \"kubernetes.io/projected/029efa89-69ea-4a94-b144-2fac56abaab4-kube-api-access-bp8cl\") pod \"managed-serviceaccount-addon-agent-54ff84c98b-2dvr8\" (UID: \"029efa89-69ea-4a94-b144-2fac56abaab4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"
Sep 18 19:39:51.303445 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.303418    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.303614 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.303455    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgwq4\" (UniqueName: \"kubernetes.io/projected/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-kube-api-access-lgwq4\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.303614 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.303485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-ca\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.303614 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.303506    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.303614 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.303531    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp8cl\" (UniqueName: \"kubernetes.io/projected/029efa89-69ea-4a94-b144-2fac56abaab4-kube-api-access-bp8cl\") pod \"managed-serviceaccount-addon-agent-54ff84c98b-2dvr8\" (UID: \"029efa89-69ea-4a94-b144-2fac56abaab4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"
Sep 18 19:39:51.303614 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.303570    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.303614 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.303605    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/029efa89-69ea-4a94-b144-2fac56abaab4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54ff84c98b-2dvr8\" (UID: \"029efa89-69ea-4a94-b144-2fac56abaab4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"
Sep 18 19:39:51.303895 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.303736    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-hub\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.304403 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.304375    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.306299 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.306275    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-ca\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.306445 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.306399    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.306525 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.306472    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.306640 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.306621    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/029efa89-69ea-4a94-b144-2fac56abaab4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54ff84c98b-2dvr8\" (UID: \"029efa89-69ea-4a94-b144-2fac56abaab4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"
Sep 18 19:39:51.306696 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.306647    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-hub\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.311253 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.311234    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgwq4\" (UniqueName: \"kubernetes.io/projected/d4ef16b8-7954-4305-bf5f-923ffd5cbeec-kube-api-access-lgwq4\") pod \"cluster-proxy-proxy-agent-6b7745c878-sqmzq\" (UID: \"d4ef16b8-7954-4305-bf5f-923ffd5cbeec\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.311322 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.311287    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp8cl\" (UniqueName: \"kubernetes.io/projected/029efa89-69ea-4a94-b144-2fac56abaab4-kube-api-access-bp8cl\") pod \"managed-serviceaccount-addon-agent-54ff84c98b-2dvr8\" (UID: \"029efa89-69ea-4a94-b144-2fac56abaab4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"
Sep 18 19:39:51.438683 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.438603    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"
Sep 18 19:39:51.453335 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.453304    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:39:51.709101 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.709075    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8"]
Sep 18 19:39:51.712096 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:39:51.712070    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod029efa89_69ea_4a94_b144_2fac56abaab4.slice/crio-9132f5c2a439a70a3b8751a2601edbbfb1100e45e0a29676537793ffd36958c1 WatchSource:0}: Error finding container 9132f5c2a439a70a3b8751a2601edbbfb1100e45e0a29676537793ffd36958c1: Status 404 returned error can't find the container with id 9132f5c2a439a70a3b8751a2601edbbfb1100e45e0a29676537793ffd36958c1
Sep 18 19:39:51.726750 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.726725    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"]
Sep 18 19:39:51.729662 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:39:51.729638    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4ef16b8_7954_4305_bf5f_923ffd5cbeec.slice/crio-3bb0abf8d1b0a545958f795b76524b73dcf4a87ed8968dffc4710f92866e3641 WatchSource:0}: Error finding container 3bb0abf8d1b0a545958f795b76524b73dcf4a87ed8968dffc4710f92866e3641: Status 404 returned error can't find the container with id 3bb0abf8d1b0a545958f795b76524b73dcf4a87ed8968dffc4710f92866e3641
Sep 18 19:39:51.773791 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.773759    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" event={"ID":"d4ef16b8-7954-4305-bf5f-923ffd5cbeec","Type":"ContainerStarted","Data":"3bb0abf8d1b0a545958f795b76524b73dcf4a87ed8968dffc4710f92866e3641"}
Sep 18 19:39:51.774856 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:51.774833    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8" event={"ID":"029efa89-69ea-4a94-b144-2fac56abaab4","Type":"ContainerStarted","Data":"9132f5c2a439a70a3b8751a2601edbbfb1100e45e0a29676537793ffd36958c1"}
Sep 18 19:39:55.789466 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:55.789433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" event={"ID":"d4ef16b8-7954-4305-bf5f-923ffd5cbeec","Type":"ContainerStarted","Data":"64a0bf54eca9981dadd7ad5dd6baecb4f08fc2a666712fcee8ef3ff045543c35"}
Sep 18 19:39:55.789893 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:55.789486    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" event={"ID":"d4ef16b8-7954-4305-bf5f-923ffd5cbeec","Type":"ContainerStarted","Data":"8dcbe820014794adc3331d7278a832e4a95189a50d3b60086ef55b096f28d862"}
Sep 18 19:39:55.791462 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:55.791433    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8" event={"ID":"029efa89-69ea-4a94-b144-2fac56abaab4","Type":"ContainerStarted","Data":"743d16f08fab4b87dde43910afa19457b9697c86fd8b365e3d788b8bad667b62"}
Sep 18 19:39:55.807185 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:55.807060    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8" podStartSLOduration=0.852301806 podStartE2EDuration="4.807045542s" totalImagesPullingTime="3.954743736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:55.80626435 +0000 UTC m=+86.768997088" watchObservedRunningTime="2026-09-18 19:39:55.807045542 +0000 UTC m=+86.769778279"
Sep 18 19:39:56.800600 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:56.800562    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" event={"ID":"d4ef16b8-7954-4305-bf5f-923ffd5cbeec","Type":"ContainerStarted","Data":"96a697dab3f0675f11a17f107e4899fd6d55e226372d5a219b4f6e7b20a02335"}
Sep 18 19:39:56.817742 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:39:56.817694    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" podStartSLOduration=1.8693503360000001 podStartE2EDuration="5.817680593s" totalImagesPullingTime="3.948330257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:39:56.816904947 +0000 UTC m=+87.779637684" watchObservedRunningTime="2026-09-18 19:39:56.817680593 +0000 UTC m=+87.780413330"
Sep 18 19:40:03.452463 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:40:03.452430    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:03.592840 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:40:03.592799    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:40:09.844264 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:40:09.844232    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:40:09.844699 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:40:09.844379    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:40:09.844699 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:40:09.844439    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:41:13.844423989 +0000 UTC m=+164.807156710 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:40:13.756953 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:40:13.756920    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-86rp7"
Sep 18 19:40:41.166221 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:40:41.166171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:40:41.166701 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:40:41.166326    2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 19:40:41.166701 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:40:41.166400    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs podName:d271732b-e2a8-4cc5-b5e6-6481da077fbb nodeName:}" failed. No retries permitted until 2026-09-18 19:42:43.166383344 +0000 UTC m=+254.129116068 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs") pod "network-metrics-daemon-vfx7n" (UID: "d271732b-e2a8-4cc5-b5e6-6481da077fbb") : secret "metrics-daemon-secret" not found
Sep 18 19:41:08.968817 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:41:08.968764    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-g792g" podUID="873ecdc4-ef5b-430d-aa6a-d0216f3344aa"
Sep 18 19:41:09.019098 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:41:09.019066    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g792g"
Sep 18 19:41:09.475289 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:41:09.475254    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-vfx7n" podUID="d271732b-e2a8-4cc5-b5e6-6481da077fbb"
Sep 18 19:41:13.886199 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:41:13.886104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:41:13.886669 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:41:13.886273    2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 19:41:13.886669 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:41:13.886358    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls podName:873ecdc4-ef5b-430d-aa6a-d0216f3344aa nodeName:}" failed. No retries permitted until 2026-09-18 19:43:15.886343345 +0000 UTC m=+286.849076061 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls") pod "dns-default-g792g" (UID: "873ecdc4-ef5b-430d-aa6a-d0216f3344aa") : secret "dns-default-metrics-tls" not found
Sep 18 19:41:20.464912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:41:20.464856    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:41:56.162362 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:41:56.162275    2559 generic.go:356] "Generic (PLEG): container finished" podID="029efa89-69ea-4a94-b144-2fac56abaab4" containerID="743d16f08fab4b87dde43910afa19457b9697c86fd8b365e3d788b8bad667b62" exitCode=255
Sep 18 19:41:56.162362 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:41:56.162334    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8" event={"ID":"029efa89-69ea-4a94-b144-2fac56abaab4","Type":"ContainerDied","Data":"743d16f08fab4b87dde43910afa19457b9697c86fd8b365e3d788b8bad667b62"}
Sep 18 19:41:56.162875 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:41:56.162616    2559 scope.go:117] "RemoveContainer" containerID="743d16f08fab4b87dde43910afa19457b9697c86fd8b365e3d788b8bad667b62"
Sep 18 19:41:57.167177 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:41:57.167142    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54ff84c98b-2dvr8" event={"ID":"029efa89-69ea-4a94-b144-2fac56abaab4","Type":"ContainerStarted","Data":"a37dd769b371880a53bad71651eda38951d262ec03bb89161002f05f197c897a"}
Sep 18 19:42:05.951808 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:05.951776    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:23.145384 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:23.145351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:23.204061 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:23.204030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:26.286293 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:26.286265    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:26.344279 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:26.344253    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:28.129106 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.129066    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6x52b"]
Sep 18 19:42:28.133292 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.133258    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.135545 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.135511    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 19:42:28.135670 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.135554    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 19:42:28.135670 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.135615    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 19:42:28.135670 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.135618    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 19:42:28.136209 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.136192    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mvsph\""
Sep 18 19:42:28.142830 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.142807    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6x52b"]
Sep 18 19:42:28.148488 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.148463    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:28.294164 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.294141    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-crio-socket\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.294247 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.294170    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.294247 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.294203    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-data-volume\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.294315 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.294263    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrkph\" (UniqueName: \"kubernetes.io/projected/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-api-access-vrkph\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.294315 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.294299    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.395206 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.395151    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-crio-socket\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.395281 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.395184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.395281 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.395244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-data-volume\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.395281 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.395264    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrkph\" (UniqueName: \"kubernetes.io/projected/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-api-access-vrkph\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.395281 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.395265    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-crio-socket\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.395471 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.395291    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.395471 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:28.395367    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:28.395471 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:28.395386    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:28.395471 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:28.395436    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:28.895418122 +0000 UTC m=+239.858150851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:28.395471 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:28.395450    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:28.895443605 +0000 UTC m=+239.858176322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:28.395712 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.395554    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-data-volume\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.403627 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.403608    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrkph\" (UniqueName: \"kubernetes.io/projected/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-api-access-vrkph\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.692134 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.692075    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:28.742266 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.742244    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:28.900219 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.900195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.900306 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.900237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:28.900345 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:28.900312    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:28.900345 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:28.900331    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:28.900405 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:28.900366    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:29.900350844 +0000 UTC m=+240.863083560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:28.900405 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:28.900379    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:29.900373225 +0000 UTC m=+240.863105941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:28.996804 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:28.996783    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:29.091672 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:29.091652    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:29.192433 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:29.192410    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:29.413450 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:29.413380    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:29.466933 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:29.466906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:29.906411 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:29.906378    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:29.906539 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:29.906450    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:29.906539 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:29.906498    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:29.906613 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:29.906568    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:31.906553472 +0000 UTC m=+242.869286188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:29.906613 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:29.906566    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:29.906613 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:29.906597    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:31.906590722 +0000 UTC m=+242.869323438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:30.101462 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:30.101440    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hvl8z_d8ab3246-2a81-4ed1-8f73-2e44beaeba85/dns-node-resolver/0.log"
Sep 18 19:42:31.035033 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:31.035002    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:31.088446 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:31.088422    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:31.100871 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:31.100850    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccfk8_0be5902d-0c5c-4c36-b955-a0d97e5e61f5/node-ca/0.log"
Sep 18 19:42:31.337632 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:31.337569    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:31.438458 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:31.438441    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:31.920477 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:31.920449    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:31.920584 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:31.920491    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:31.920584 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:31.920558    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:31.920584 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:31.920573    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:31.920697 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:31.920610    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:35.920595917 +0000 UTC m=+246.883328633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:31.920697 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:31.920625    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:35.920617673 +0000 UTC m=+246.883350389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:31.937651 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:31.937625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:32.101590 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:32.101568    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fpsxk_789fc76c-eb30-45d8-9e8f-a6ac1f6fb324/serve-healthcheck-canary/0.log"
Sep 18 19:42:33.419870 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:33.419833    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:33.584377 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:33.584343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:33.685313 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:33.685240    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:33.852262 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:33.852229    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:33.952434 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:33.952405    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:34.017841 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:34.017817    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:34.127093 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:34.127068    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:34.391334 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:34.391271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:35.312909 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:35.312874    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:35.515249 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:35.515223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:35.714570 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:35.714542    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:35.929555 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:35.929528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:35.949155 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:35.949134    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:35.949214 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:35.949177    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:35.949277 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:35.949259    2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 19:42:35.949314 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:35.949290    2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:35.949347 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:35.949335    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:43.949315879 +0000 UTC m=+254.912048614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : configmap "kube-rbac-proxy" not found
Sep 18 19:42:35.949392 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:42:35.949349    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls podName:560d1f47-6808-41b0-b1ac-f0efdd6cdd16 nodeName:}" failed. No retries permitted until 2026-09-18 19:42:43.949343317 +0000 UTC m=+254.912076032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6x52b" (UID: "560d1f47-6808-41b0-b1ac-f0efdd6cdd16") : secret "insights-runtime-extractor-tls" not found
Sep 18 19:42:36.115159 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:36.115091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:36.326666 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:36.326645    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.114555 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:37.114532    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.299184 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:37.299161    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.354097 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:37.354073    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.517768 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:37.517743    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.723885 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:37.723859    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.824699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:37.824648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.924819 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:37.924796    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:37.989331 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:37.989308    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:38.118635 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:38.118574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:39.317060 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:39.317028    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:39.520440 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:39.520411    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:39.716506 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:39.716481    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:39.908212 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:39.908157    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:39.963743 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:39.963719    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:40.113384 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:40.113310    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:41.117526 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:41.117502    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:41.408583 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:41.408526    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:41.514719 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:41.514688    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:41.717127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:41.717105    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:41.920569 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:41.920541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:42.314546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:42.314520    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:43.113999 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.113953    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:43.201347 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.201311    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:42:43.203819 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.203795    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d271732b-e2a8-4cc5-b5e6-6481da077fbb-metrics-certs\") pod \"network-metrics-daemon-vfx7n\" (UID: \"d271732b-e2a8-4cc5-b5e6-6481da077fbb\") " pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:42:43.268265 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.268243    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nv7t\""
Sep 18 19:42:43.277118 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.277098    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vfx7n"
Sep 18 19:42:43.313451 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.313425    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:43.515000 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.514956    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:43.544047 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.544028    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vfx7n"]
Sep 18 19:42:43.547103 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:42:43.547071    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd271732b_e2a8_4cc5_b5e6_6481da077fbb.slice/crio-831bd685e4e61af72f1ea57ee7f21dfd0717a8c11061eec8c540acc1defd7a1d WatchSource:0}: Error finding container 831bd685e4e61af72f1ea57ee7f21dfd0717a8c11061eec8c540acc1defd7a1d: Status 404 returned error can't find the container with id 831bd685e4e61af72f1ea57ee7f21dfd0717a8c11061eec8c540acc1defd7a1d
Sep 18 19:42:43.911512 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:43.911459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:44.006036 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.006012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:44.006118 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.006071    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:44.006608 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.006587    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:44.008275 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.008259    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/560d1f47-6808-41b0-b1ac-f0efdd6cdd16-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6x52b\" (UID: \"560d1f47-6808-41b0-b1ac-f0efdd6cdd16\") " pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:44.045361 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.045338    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mvsph\""
Sep 18 19:42:44.053951 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.053934    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6x52b"
Sep 18 19:42:44.111514 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.111491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:44.308408 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.308381    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfx7n" event={"ID":"d271732b-e2a8-4cc5-b5e6-6481da077fbb","Type":"ContainerStarted","Data":"831bd685e4e61af72f1ea57ee7f21dfd0717a8c11061eec8c540acc1defd7a1d"}
Sep 18 19:42:44.323450 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:44.323426    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6x52b"]
Sep 18 19:42:44.326842 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:42:44.326817    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560d1f47_6808_41b0_b1ac_f0efdd6cdd16.slice/crio-34fec89dd27e797f01d2f461d8f6595e6544f2206bb1ae7a1d80080aa5d0c11c WatchSource:0}: Error finding container 34fec89dd27e797f01d2f461d8f6595e6544f2206bb1ae7a1d80080aa5d0c11c: Status 404 returned error can't find the container with id 34fec89dd27e797f01d2f461d8f6595e6544f2206bb1ae7a1d80080aa5d0c11c
Sep 18 19:42:45.316071 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:45.316002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfx7n" event={"ID":"d271732b-e2a8-4cc5-b5e6-6481da077fbb","Type":"ContainerStarted","Data":"dedcdde5fdb34198b82eacd6cfb9d43e1a6e2155e19f6f1dcbaedeb536f7f7de"}
Sep 18 19:42:45.316456 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:45.316109    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vfx7n" event={"ID":"d271732b-e2a8-4cc5-b5e6-6481da077fbb","Type":"ContainerStarted","Data":"8bec3d7fc2e4cf4528071ce91b34759b822fb9b2f7c6808d74f053af50b824f6"}
Sep 18 19:42:45.317908 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:45.317878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6x52b" event={"ID":"560d1f47-6808-41b0-b1ac-f0efdd6cdd16","Type":"ContainerStarted","Data":"53cb0c2e91700d2448705c1923c76529a95f5772f8d493067819b82307b4bb21"}
Sep 18 19:42:45.318032 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:45.317926    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6x52b" event={"ID":"560d1f47-6808-41b0-b1ac-f0efdd6cdd16","Type":"ContainerStarted","Data":"fd0a70a2b43ff730e0965c76b2a68ea9837066304a6c89d5b1b3ee62309bc85c"}
Sep 18 19:42:45.318032 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:45.317947    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6x52b" event={"ID":"560d1f47-6808-41b0-b1ac-f0efdd6cdd16","Type":"ContainerStarted","Data":"34fec89dd27e797f01d2f461d8f6595e6544f2206bb1ae7a1d80080aa5d0c11c"}
Sep 18 19:42:45.329950 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:45.329898    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vfx7n" podStartSLOduration=253.226900298 podStartE2EDuration="4m14.329881618s" totalImagesPullingTime="1.10298132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:38:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:45.329197523 +0000 UTC m=+256.291930301" watchObservedRunningTime="2026-09-18 19:42:45.329881618 +0000 UTC m=+256.292614352"
Sep 18 19:42:47.327365 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:47.327329    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6x52b" event={"ID":"560d1f47-6808-41b0-b1ac-f0efdd6cdd16","Type":"ContainerStarted","Data":"5fa8a852edce635cee15521be44e2ca32b4f6ef41d6707f9b9570f5ca1580c81"}
Sep 18 19:42:47.345258 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:47.345211    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6x52b" podStartSLOduration=17.384465219 podStartE2EDuration="19.345194918s" totalImagesPullingTime="1.960729699s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 19:42:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:47.344750848 +0000 UTC m=+258.307483626" watchObservedRunningTime="2026-09-18 19:42:47.345194918 +0000 UTC m=+258.307927660"
Sep 18 19:42:51.445098 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.445062    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8444bdf559-88d46"]
Sep 18 19:42:51.448453 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.448425    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.451240 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.451214    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rwqd5\""
Sep 18 19:42:51.451360 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.451214    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 19:42:51.451593 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.451575    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 19:42:51.451760 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.451740    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 19:42:51.463531 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.463506    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 19:42:51.463718 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.463693    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8444bdf559-88d46"]
Sep 18 19:42:51.510221 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.510183    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:51.557023 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.557001    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25630fa6-a5c9-41b5-a564-2a15445ba92f-ca-trust-extracted\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.557117 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.557034    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25630fa6-a5c9-41b5-a564-2a15445ba92f-trusted-ca\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.557117 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.557064    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25630fa6-a5c9-41b5-a564-2a15445ba92f-installation-pull-secrets\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.557117 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.557085    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-bound-sa-token\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.557224 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.557124    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25630fa6-a5c9-41b5-a564-2a15445ba92f-image-registry-private-configuration\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.557224 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.557139    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-registry-tls\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.557224 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.557165    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25630fa6-a5c9-41b5-a564-2a15445ba92f-registry-certificates\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.557224 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.557180    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6sh\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-kube-api-access-mj6sh\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.608711 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.608690    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:51.658387 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.658358    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25630fa6-a5c9-41b5-a564-2a15445ba92f-ca-trust-extracted\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.658497 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.658390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25630fa6-a5c9-41b5-a564-2a15445ba92f-trusted-ca\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.658497 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.658425    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25630fa6-a5c9-41b5-a564-2a15445ba92f-installation-pull-secrets\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.658497 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.658440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-bound-sa-token\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.658497 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.658472    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25630fa6-a5c9-41b5-a564-2a15445ba92f-image-registry-private-configuration\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.658688 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.658497    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-registry-tls\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.658688 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.658534    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25630fa6-a5c9-41b5-a564-2a15445ba92f-registry-certificates\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.658688 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.658557    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6sh\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-kube-api-access-mj6sh\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.659074 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.659054    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25630fa6-a5c9-41b5-a564-2a15445ba92f-ca-trust-extracted\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.659561 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.659533    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25630fa6-a5c9-41b5-a564-2a15445ba92f-trusted-ca\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.659561 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.659547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25630fa6-a5c9-41b5-a564-2a15445ba92f-registry-certificates\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.661312 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.661290    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25630fa6-a5c9-41b5-a564-2a15445ba92f-installation-pull-secrets\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.661407 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.661327    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25630fa6-a5c9-41b5-a564-2a15445ba92f-image-registry-private-configuration\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.661553 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.661528    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-registry-tls\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.669505 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.669478    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-bound-sa-token\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.669602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.669511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6sh\" (UniqueName: \"kubernetes.io/projected/25630fa6-a5c9-41b5-a564-2a15445ba92f-kube-api-access-mj6sh\") pod \"image-registry-8444bdf559-88d46\" (UID: \"25630fa6-a5c9-41b5-a564-2a15445ba92f\") " pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:51.764075 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:51.764044    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:52.036987 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:52.036908    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8444bdf559-88d46"]
Sep 18 19:42:52.039790 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:42:52.039761    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25630fa6_a5c9_41b5_a564_2a15445ba92f.slice/crio-f5aa4cc63822942279377b83952a9fe21fcab084f58c7411c382d9e5ce36179d WatchSource:0}: Error finding container f5aa4cc63822942279377b83952a9fe21fcab084f58c7411c382d9e5ce36179d: Status 404 returned error can't find the container with id f5aa4cc63822942279377b83952a9fe21fcab084f58c7411c382d9e5ce36179d
Sep 18 19:42:52.346827 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:52.346755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8444bdf559-88d46" event={"ID":"25630fa6-a5c9-41b5-a564-2a15445ba92f","Type":"ContainerStarted","Data":"35186c8506a2add7bb44163ed12b2376c8e9ae2baaf1b68dd4945e5f636daba1"}
Sep 18 19:42:52.346827 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:52.346812    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8444bdf559-88d46" event={"ID":"25630fa6-a5c9-41b5-a564-2a15445ba92f","Type":"ContainerStarted","Data":"f5aa4cc63822942279377b83952a9fe21fcab084f58c7411c382d9e5ce36179d"}
Sep 18 19:42:52.347035 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:52.346863    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:42:52.364192 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:52.364146    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8444bdf559-88d46" podStartSLOduration=1.364131617 podStartE2EDuration="1.364131617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:42:52.363180079 +0000 UTC m=+263.325912818" watchObservedRunningTime="2026-09-18 19:42:52.364131617 +0000 UTC m=+263.326864355"
Sep 18 19:42:52.474532 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:52.474500    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:52.530249 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:52.530223    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:57.230300 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:57.230256    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:57.328013 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:57.327980    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:57.396155 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:57.396132    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:42:57.504798 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:42:57.504737    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:01.417022 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:01.416975    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:01.485800 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:01.485773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:01.565389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:01.565364    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:03.818841 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:03.818797    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:04.892677 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:04.892638    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:05.557555 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.557521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:05.787319 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.787289    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:05.838424 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.838355    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-whlb4"]
Sep 18 19:43:05.843247 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.843229    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.845646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.845591    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 19:43:05.845646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.845608    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jkr4l\""
Sep 18 19:43:05.845856 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.845843    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 19:43:05.846338 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.846320    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 19:43:05.846442 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.846320    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 19:43:05.846442 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.846354    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 19:43:05.846442 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.846337    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 19:43:05.884170 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.884151    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:05.952285 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952256    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e51ef1-9886-44bc-be5a-c04428bbb52d-metrics-client-ca\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.952546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952303    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wldts\" (UniqueName: \"kubernetes.io/projected/e5e51ef1-9886-44bc-be5a-c04428bbb52d-kube-api-access-wldts\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.952546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952329    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-sys\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.952546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-wtmp\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.952546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952390    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-textfile\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.952546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952410    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-accelerators-collector-config\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.952546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952452    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-root\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.952546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952480    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-tls\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.952546 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.952498    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:05.984355 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:05.984327    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:06.052910 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.052888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e51ef1-9886-44bc-be5a-c04428bbb52d-metrics-client-ca\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.052932    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wldts\" (UniqueName: \"kubernetes.io/projected/e5e51ef1-9886-44bc-be5a-c04428bbb52d-kube-api-access-wldts\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.052954    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-sys\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053019    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-sys\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053190 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053106    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-wtmp\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053190 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053130    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-textfile\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053190 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-accelerators-collector-config\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053190 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053175    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-root\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053386 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053195    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-tls\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053386 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053225    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053386 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053273    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-root\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053386 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053310    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-wtmp\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053386 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:43:06.053378    2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 19:43:06.053727 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-textfile\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053727 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:43:06.053454    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-tls podName:e5e51ef1-9886-44bc-be5a-c04428bbb52d nodeName:}" failed. No retries permitted until 2026-09-18 19:43:06.553435077 +0000 UTC m=+277.516167817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-tls") pod "node-exporter-whlb4" (UID: "e5e51ef1-9886-44bc-be5a-c04428bbb52d") : secret "node-exporter-tls" not found
Sep 18 19:43:06.053727 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053477    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e5e51ef1-9886-44bc-be5a-c04428bbb52d-metrics-client-ca\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.053727 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.053692    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-accelerators-collector-config\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.055584 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.055558    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.061326 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.061303    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wldts\" (UniqueName: \"kubernetes.io/projected/e5e51ef1-9886-44bc-be5a-c04428bbb52d-kube-api-access-wldts\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.084577 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.084556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:06.186409 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.186352    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:06.286596 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.286574    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:06.386925 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.386904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:06.556667 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.556644    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-tls\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.558833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.558808    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e5e51ef1-9886-44bc-be5a-c04428bbb52d-node-exporter-tls\") pod \"node-exporter-whlb4\" (UID: \"e5e51ef1-9886-44bc-be5a-c04428bbb52d\") " pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.752136 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.752115    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-whlb4"
Sep 18 19:43:06.764505 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:43:06.764481    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e51ef1_9886_44bc_be5a_c04428bbb52d.slice/crio-6479cff26255d038040cc02b67bc0ff9784fa6ba03e25b0ce9fd8eae84fc4e7b WatchSource:0}: Error finding container 6479cff26255d038040cc02b67bc0ff9784fa6ba03e25b0ce9fd8eae84fc4e7b: Status 404 returned error can't find the container with id 6479cff26255d038040cc02b67bc0ff9784fa6ba03e25b0ce9fd8eae84fc4e7b
Sep 18 19:43:06.803072 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.803053    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:06.884331 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:06.884277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:07.005096 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:07.005073    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:07.059040 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:07.059014    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:07.395215 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:07.395190    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-whlb4" event={"ID":"e5e51ef1-9886-44bc-be5a-c04428bbb52d","Type":"ContainerStarted","Data":"6479cff26255d038040cc02b67bc0ff9784fa6ba03e25b0ce9fd8eae84fc4e7b"}
Sep 18 19:43:07.920372 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:07.920346    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:08.400447 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:08.400410    2559 generic.go:356] "Generic (PLEG): container finished" podID="e5e51ef1-9886-44bc-be5a-c04428bbb52d" containerID="2bacd912f8ef4ebed9ac8b759941150e89e8296ee2d98950006ca4aaa4ad274c" exitCode=0
Sep 18 19:43:08.400790 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:08.400464    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-whlb4" event={"ID":"e5e51ef1-9886-44bc-be5a-c04428bbb52d","Type":"ContainerDied","Data":"2bacd912f8ef4ebed9ac8b759941150e89e8296ee2d98950006ca4aaa4ad274c"}
Sep 18 19:43:08.836773 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:08.836740    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:08.905566 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:08.905540    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:08.993635 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:08.993611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:09.408786 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:09.408755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-whlb4" event={"ID":"e5e51ef1-9886-44bc-be5a-c04428bbb52d","Type":"ContainerStarted","Data":"52644cc5bfdf159aea5502c24fde968d5cae988dbab4324cc338aef142c2713a"}
Sep 18 19:43:09.409171 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:09.408815    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-whlb4" event={"ID":"e5e51ef1-9886-44bc-be5a-c04428bbb52d","Type":"ContainerStarted","Data":"f7d6c6fa4fa575db631f8a8c4a176506b4d08d1269dd7b8d5ea730f921d3e184"}
Sep 18 19:43:09.434895 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:09.434824    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-whlb4" podStartSLOduration=3.346467502 podStartE2EDuration="4.434807688s" totalImagesPullingTime="1.027866129s" totalInitContainerRuntime="60.474057ms" isStatefulPod=true podCreationTimestamp="2026-09-18 19:43:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:09.433060458 +0000 UTC m=+280.395793198" watchObservedRunningTime="2026-09-18 19:43:09.434807688 +0000 UTC m=+280.397540480"
Sep 18 19:43:09.927106 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:09.927072    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:10.586101 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:10.586069    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:10.798374 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:10.798342    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:10.961656 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:10.961625    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:11.021768 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.021738    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"]
Sep 18 19:43:11.024968 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.024940    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.027245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.027205    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 19:43:11.027379 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.027220    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 19:43:11.027379 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.027220    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 19:43:11.027379 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.027248    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 19:43:11.027981 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.027944    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 19:43:11.032386 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.028574    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-n2hrs\""
Sep 18 19:43:11.037794 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.037767    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 19:43:11.048538 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.048513    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"]
Sep 18 19:43:11.079077 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.079048    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:11.083539 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.083513    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-serving-certs-ca-bundle\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.083647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.083544    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.083647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.083565    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-federate-client-tls\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.083647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.083607    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-secret-telemeter-client\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.083647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.083634    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.083802 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.083681    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-telemeter-client-tls\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.083802 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.083713    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ml72\" (UniqueName: \"kubernetes.io/projected/431a45a4-acb9-495d-a904-3691e3a21c72-kube-api-access-7ml72\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.083802 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.083742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-metrics-client-ca\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.160909 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.160884    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:11.183945 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.183920    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ml72\" (UniqueName: \"kubernetes.io/projected/431a45a4-acb9-495d-a904-3691e3a21c72-kube-api-access-7ml72\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184047 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.183949    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-metrics-client-ca\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184047 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184004    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-serving-certs-ca-bundle\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184047 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184035    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184184 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184065    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-federate-client-tls\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184184 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184149    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-secret-telemeter-client\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184308 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184190    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184308 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184250    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-telemeter-client-tls\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184790 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184758    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-metrics-client-ca\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.184942 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184920    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.185057 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.184987    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/431a45a4-acb9-495d-a904-3691e3a21c72-serving-certs-ca-bundle\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.186724 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.186705    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.187012 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.186991    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-secret-telemeter-client\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.187183 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.187167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-telemeter-client-tls\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.187248 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.187173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/431a45a4-acb9-495d-a904-3691e3a21c72-federate-client-tls\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.191362 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.191337    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ml72\" (UniqueName: \"kubernetes.io/projected/431a45a4-acb9-495d-a904-3691e3a21c72-kube-api-access-7ml72\") pod \"telemeter-client-7f59bdcb78-mwxcj\" (UID: \"431a45a4-acb9-495d-a904-3691e3a21c72\") " pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.338576 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.338530    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"
Sep 18 19:43:11.609708 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:11.609633    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj"]
Sep 18 19:43:11.613986 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:43:11.613950    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431a45a4_acb9_495d_a904_3691e3a21c72.slice/crio-de6c7814a77e6ca2cccc0db76947e574f81f6e39b52df78f39fcdc09ed4dc4df WatchSource:0}: Error finding container de6c7814a77e6ca2cccc0db76947e574f81f6e39b52df78f39fcdc09ed4dc4df: Status 404 returned error can't find the container with id de6c7814a77e6ca2cccc0db76947e574f81f6e39b52df78f39fcdc09ed4dc4df
Sep 18 19:43:12.019558 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:43:12.019524    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-g792g" podUID="873ecdc4-ef5b-430d-aa6a-d0216f3344aa"
Sep 18 19:43:12.149039 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:12.149011    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:12.419761 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:12.419690    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g792g"
Sep 18 19:43:12.419761 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:12.419708    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj" event={"ID":"431a45a4-acb9-495d-a904-3691e3a21c72","Type":"ContainerStarted","Data":"de6c7814a77e6ca2cccc0db76947e574f81f6e39b52df78f39fcdc09ed4dc4df"}
Sep 18 19:43:13.313937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:13.313902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:13.356303 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:13.356277    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8444bdf559-88d46"
Sep 18 19:43:13.410887 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:13.410854    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:13.488835 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:13.488810    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:13.590649 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:13.590568    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:13.647577 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:13.647541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:14.427254 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:14.427230    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj" event={"ID":"431a45a4-acb9-495d-a904-3691e3a21c72","Type":"ContainerStarted","Data":"bc692fd99bf8bd5b5b3be714d134d1b70ab209f6623b4de99a187ab56fd89068"}
Sep 18 19:43:14.427522 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:14.427280    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj" event={"ID":"431a45a4-acb9-495d-a904-3691e3a21c72","Type":"ContainerStarted","Data":"b18158d6ddd6f35c0cc0d12817cf8b9d62e7b6d8b3bc9b39879660d2a44ad941"}
Sep 18 19:43:15.434319 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:15.434279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj" event={"ID":"431a45a4-acb9-495d-a904-3691e3a21c72","Type":"ContainerStarted","Data":"d0d147fd64f9ff5566538984091dec8e4e61376ef49eb5eda3d4f74631929794"}
Sep 18 19:43:15.456861 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:15.456803    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f59bdcb78-mwxcj" podStartSLOduration=2.787131224 podStartE2EDuration="5.456785326s" totalImagesPullingTime="2.669654102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:15.455427118 +0000 UTC m=+286.418159856" watchObservedRunningTime="2026-09-18 19:43:15.456785326 +0000 UTC m=+286.419518065"
Sep 18 19:43:15.913933 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:15.913902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:43:15.916391 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:15.916368    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/873ecdc4-ef5b-430d-aa6a-d0216f3344aa-metrics-tls\") pod \"dns-default-g792g\" (UID: \"873ecdc4-ef5b-430d-aa6a-d0216f3344aa\") " pod="openshift-dns/dns-default-g792g"
Sep 18 19:43:16.022631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:16.022604    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-j9gtw\""
Sep 18 19:43:16.031378 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:16.031352    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g792g"
Sep 18 19:43:16.303416 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:16.303391    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g792g"]
Sep 18 19:43:16.306065 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:43:16.306028    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873ecdc4_ef5b_430d_aa6a_d0216f3344aa.slice/crio-1cc721645cdec2b0ae09e3e1f324d4ace8ee8098d20380c59adfd6dbd55be915 WatchSource:0}: Error finding container 1cc721645cdec2b0ae09e3e1f324d4ace8ee8098d20380c59adfd6dbd55be915: Status 404 returned error can't find the container with id 1cc721645cdec2b0ae09e3e1f324d4ace8ee8098d20380c59adfd6dbd55be915
Sep 18 19:43:16.439018 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:16.438981    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g792g" event={"ID":"873ecdc4-ef5b-430d-aa6a-d0216f3344aa","Type":"ContainerStarted","Data":"1cc721645cdec2b0ae09e3e1f324d4ace8ee8098d20380c59adfd6dbd55be915"}
Sep 18 19:43:18.447696 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:18.447623    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g792g" event={"ID":"873ecdc4-ef5b-430d-aa6a-d0216f3344aa","Type":"ContainerStarted","Data":"cff417ae64dae797f92e21f01d3ca857b013d62e2b45ebcca892737233e8987d"}
Sep 18 19:43:18.447696 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:18.447689    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g792g" event={"ID":"873ecdc4-ef5b-430d-aa6a-d0216f3344aa","Type":"ContainerStarted","Data":"c3bf7f493da9a5644b77e1045f7080dc5f422d2bfd370b78d9f38aefe067d1b2"}
Sep 18 19:43:18.448077 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:18.447772    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g792g"
Sep 18 19:43:18.464409 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:18.464369    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g792g" podStartSLOduration=251.623055362 podStartE2EDuration="4m13.464355248s" totalImagesPullingTime="1.841299886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:39:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:18.464056885 +0000 UTC m=+289.426789625" watchObservedRunningTime="2026-09-18 19:43:18.464355248 +0000 UTC m=+289.427087986"
Sep 18 19:43:20.787932 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:20.787876    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:20.888720 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:20.888691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:20.944898 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:20.944867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.098438 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:21.098369    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.269509 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:21.269482    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.324859 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:21.324829    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.684481 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:21.684448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.743702 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:21.743671    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:21.867664 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:21.867638    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:22.289444 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:22.289412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:22.352581 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:22.352556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:22.472669 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:22.472649    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:22.525111 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:22.525087    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:22.673128 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:22.673061    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:23.141634 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:23.141598    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:26.160137 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.160095    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:26.252129 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.252101    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:26.674381 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.674352    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f999485bf-qtbc9"]
Sep 18 19:43:26.679369 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.679351    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.681664 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.681638    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 19:43:26.682186 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.682164    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 19:43:26.682334 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.682306    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 19:43:26.682334 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.682326    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 19:43:26.682465 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.682360    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7m6z5\""
Sep 18 19:43:26.682465 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.682326    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 19:43:26.682465 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.682369    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 19:43:26.682465 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.682377    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 19:43:26.688357 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.688330    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f999485bf-qtbc9"]
Sep 18 19:43:26.785190 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.785162    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-service-ca\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.785295 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.785198    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-serving-cert\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.785295 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.785221    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-oauth-config\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.785295 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.785244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-config\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.785419 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.785324    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjfv\" (UniqueName: \"kubernetes.io/projected/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-kube-api-access-qfjfv\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.785419 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.785353    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-oauth-serving-cert\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.886411 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.886381    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjfv\" (UniqueName: \"kubernetes.io/projected/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-kube-api-access-qfjfv\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.886514 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.886415    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-oauth-serving-cert\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.886514 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.886440    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-service-ca\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.886514 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.886464    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-serving-cert\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.886677 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.886574    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-oauth-config\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.886677 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.886620    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-config\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.887248 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.887221    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-oauth-serving-cert\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.887361 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.887340    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-config\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.887413 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.887344    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-service-ca\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.889105 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.889083    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-oauth-config\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.889241 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.889222    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-serving-cert\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.893938 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.893914    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjfv\" (UniqueName: \"kubernetes.io/projected/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-kube-api-access-qfjfv\") pod \"console-f999485bf-qtbc9\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") " pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:26.989941 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:26.989913    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:27.264716 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:27.264654    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f999485bf-qtbc9"]
Sep 18 19:43:27.480053 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:27.480027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f999485bf-qtbc9" event={"ID":"ce22b80e-70cd-4ad6-b562-ca79cbe6e480","Type":"ContainerStarted","Data":"7b302ec7e669df1cab2710b1f6ae4f6efa89099765d3897ac573173fd06e0b0c"}
Sep 18 19:43:28.454530 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:28.454407    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g792g"
Sep 18 19:43:30.549763 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:30.549718    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:30.650639 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:30.650605    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:31.331137 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:31.331116    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:43:31.333310 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:31.333295    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:43:31.336232 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:31.336213    2559 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 19:43:31.457169 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:31.457124    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" podUID="d4ef16b8-7954-4305-bf5f-923ffd5cbeec" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:43:31.497075 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:31.497046    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f999485bf-qtbc9" event={"ID":"ce22b80e-70cd-4ad6-b562-ca79cbe6e480","Type":"ContainerStarted","Data":"e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02"}
Sep 18 19:43:31.514359 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:31.514320    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f999485bf-qtbc9" podStartSLOduration=1.502021973 podStartE2EDuration="5.514309007s" totalImagesPullingTime="4.012287034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:43:31.513031247 +0000 UTC m=+302.475763988" watchObservedRunningTime="2026-09-18 19:43:31.514309007 +0000 UTC m=+302.477041745"
Sep 18 19:43:32.086647 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:32.086616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:32.137644 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:32.137613    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:33.691642 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:33.691607    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:36.990715 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:36.990680    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:36.991315 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:36.990748    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:36.995465 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:36.995437    2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:37.296104 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:37.296026    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:37.351277 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:37.351254    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:37.482526 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:37.482501    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:37.527286 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:37.527262    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:43:37.547580 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:37.547521    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:39.566374 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:39.566343    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:39.655304 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:39.655277    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:41.454864 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:41.454824    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" podUID="d4ef16b8-7954-4305-bf5f-923ffd5cbeec" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:43:46.031471 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:46.031433    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f999485bf-qtbc9"]
Sep 18 19:43:46.902611 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:46.902579    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:46.970303 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:46.970271    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:47.919946 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:47.919917    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:51.454750 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:51.454713    2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" podUID="d4ef16b8-7954-4305-bf5f-923ffd5cbeec" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 18 19:43:51.455126 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:51.454784    2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq"
Sep 18 19:43:51.455226 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:51.455195    2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"96a697dab3f0675f11a17f107e4899fd6d55e226372d5a219b4f6e7b20a02335"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 18 19:43:51.455262 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:51.455244    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" podUID="d4ef16b8-7954-4305-bf5f-923ffd5cbeec" containerName="service-proxy" containerID="cri-o://96a697dab3f0675f11a17f107e4899fd6d55e226372d5a219b4f6e7b20a02335" gracePeriod=30
Sep 18 19:43:51.570441 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:51.570419    2559 generic.go:356] "Generic (PLEG): container finished" podID="d4ef16b8-7954-4305-bf5f-923ffd5cbeec" containerID="96a697dab3f0675f11a17f107e4899fd6d55e226372d5a219b4f6e7b20a02335" exitCode=2
Sep 18 19:43:51.570525 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:51.570456    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" event={"ID":"d4ef16b8-7954-4305-bf5f-923ffd5cbeec","Type":"ContainerDied","Data":"96a697dab3f0675f11a17f107e4899fd6d55e226372d5a219b4f6e7b20a02335"}
Sep 18 19:43:52.473356 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:52.473319    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:52.527938 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:52.527911    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:43:52.576451 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:43:52.576415    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b7745c878-sqmzq" event={"ID":"d4ef16b8-7954-4305-bf5f-923ffd5cbeec","Type":"ContainerStarted","Data":"735b5483911d0a29f19a992c9840fe8c8d8188110f6f75ef822027d0e655c1ce"}
Sep 18 19:44:07.478972 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:07.478926    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:07.539225 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:07.539192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:11.050627 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.050588    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f999485bf-qtbc9" podUID="ce22b80e-70cd-4ad6-b562-ca79cbe6e480" containerName="console" containerID="cri-o://e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02" gracePeriod=15
Sep 18 19:44:11.286324 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.286300    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:44:11.383551 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.383461    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfjfv\" (UniqueName: \"kubernetes.io/projected/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-kube-api-access-qfjfv\") pod \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") "
Sep 18 19:44:11.383551 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.383509    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-serving-cert\") pod \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") "
Sep 18 19:44:11.383782 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.383559    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-config\") pod \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") "
Sep 18 19:44:11.383782 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.383590    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-service-ca\") pod \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") "
Sep 18 19:44:11.383782 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.383640    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-oauth-config\") pod \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") "
Sep 18 19:44:11.383782 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.383674    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-oauth-serving-cert\") pod \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\" (UID: \"ce22b80e-70cd-4ad6-b562-ca79cbe6e480\") "
Sep 18 19:44:11.384022 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.383989    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-config" (OuterVolumeSpecName: "console-config") pod "ce22b80e-70cd-4ad6-b562-ca79cbe6e480" (UID: "ce22b80e-70cd-4ad6-b562-ca79cbe6e480"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:11.384146 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.384075    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-service-ca" (OuterVolumeSpecName: "service-ca") pod "ce22b80e-70cd-4ad6-b562-ca79cbe6e480" (UID: "ce22b80e-70cd-4ad6-b562-ca79cbe6e480"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:11.384254 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.384142    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ce22b80e-70cd-4ad6-b562-ca79cbe6e480" (UID: "ce22b80e-70cd-4ad6-b562-ca79cbe6e480"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:44:11.385775 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.385757    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ce22b80e-70cd-4ad6-b562-ca79cbe6e480" (UID: "ce22b80e-70cd-4ad6-b562-ca79cbe6e480"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:11.386127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.386106    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ce22b80e-70cd-4ad6-b562-ca79cbe6e480" (UID: "ce22b80e-70cd-4ad6-b562-ca79cbe6e480"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:44:11.386213 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.386169    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-kube-api-access-qfjfv" (OuterVolumeSpecName: "kube-api-access-qfjfv") pod "ce22b80e-70cd-4ad6-b562-ca79cbe6e480" (UID: "ce22b80e-70cd-4ad6-b562-ca79cbe6e480"). InnerVolumeSpecName "kube-api-access-qfjfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:44:11.484804 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.484782    2559 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-oauth-serving-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:44:11.484804 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.484802    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjfv\" (UniqueName: \"kubernetes.io/projected/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-kube-api-access-qfjfv\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:44:11.484945 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.484811    2559 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-serving-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:44:11.484945 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.484819    2559 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:44:11.484945 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.484828    2559 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-service-ca\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:44:11.484945 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.484836    2559 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ce22b80e-70cd-4ad6-b562-ca79cbe6e480-console-oauth-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:44:11.639707 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.639626    2559 generic.go:356] "Generic (PLEG): container finished" podID="ce22b80e-70cd-4ad6-b562-ca79cbe6e480" containerID="e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02" exitCode=0
Sep 18 19:44:11.639707 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.639668    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f999485bf-qtbc9"
Sep 18 19:44:11.639900 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.639667    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f999485bf-qtbc9" event={"ID":"ce22b80e-70cd-4ad6-b562-ca79cbe6e480","Type":"ContainerDied","Data":"e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02"}
Sep 18 19:44:11.639900 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.639795    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f999485bf-qtbc9" event={"ID":"ce22b80e-70cd-4ad6-b562-ca79cbe6e480","Type":"ContainerDied","Data":"7b302ec7e669df1cab2710b1f6ae4f6efa89099765d3897ac573173fd06e0b0c"}
Sep 18 19:44:11.639900 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.639821    2559 scope.go:117] "RemoveContainer" containerID="e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02"
Sep 18 19:44:11.649081 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.649060    2559 scope.go:117] "RemoveContainer" containerID="e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02"
Sep 18 19:44:11.649483 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:44:11.649464    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02\": container with ID starting with e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02 not found: ID does not exist" containerID="e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02"
Sep 18 19:44:11.649563 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.649490    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02"} err="failed to get container status \"e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02\": rpc error: code = NotFound desc = could not find container \"e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02\": container with ID starting with e409bbc17bcd69172bf8d15c6993740b2dca46b0931929da0429f094cbac0d02 not found: ID does not exist"
Sep 18 19:44:11.654142 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.654122    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f999485bf-qtbc9"]
Sep 18 19:44:11.660037 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:11.660016    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f999485bf-qtbc9"]
Sep 18 19:44:13.469815 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:13.469725    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce22b80e-70cd-4ad6-b562-ca79cbe6e480" path="/var/lib/kubelet/pods/ce22b80e-70cd-4ad6-b562-ca79cbe6e480/volumes"
Sep 18 19:44:16.213128 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:16.213091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:16.278644 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:16.278588    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:21.987819 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:21.987771    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:22.053369 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:22.053339    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:33.891567 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:33.891529    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:33.992030 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:33.991991    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.240486 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:37.240448    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.292173 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:37.292131    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.484264 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:37.484225    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:37.585480 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:37.585396    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:40.950570 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:40.950528    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:41.050936 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:41.050906    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:48.407958 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:48.407930    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:48.460182 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:48.460156    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:48.515523 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:48.515491    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:44:48.616699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:44:48.616660    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:10.171320 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:10.171273    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:10.270395 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:10.270360    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:10.324575 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:10.324541    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:24.637781 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:24.637750    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:24.702277 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:24.702241    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:32.350004 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:32.349950    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:32.449673 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:32.449633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:43.005120 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:43.005019    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:43.293259 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:43.293175    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:43.517607 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:43.517581    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:43.625888 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:43.625808    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:43.725056 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:43.725030    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:44.354742 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:44.354691    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:44.452313 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:44.452284    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:44.551789 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:44.551756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:44.651416 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:44.651344    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:44.776877 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:44.776855    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:44.952727 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:44.952699    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:45.077792 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:45.077760    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:45:45.176926 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:45:45.176896    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:36.214603 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:46:36.214566    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:36.263726 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:46:36.263680    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:53.368741 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:46:53.368709    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:46:57.414413 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:46:57.414375    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.524598 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.524556    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.614443 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.614406    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.621430 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.621408    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-v5vb4"]
Sep 18 19:47:00.621745 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.621728    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce22b80e-70cd-4ad6-b562-ca79cbe6e480" containerName="console"
Sep 18 19:47:00.621787 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.621747    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce22b80e-70cd-4ad6-b562-ca79cbe6e480" containerName="console"
Sep 18 19:47:00.621820 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.621810    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce22b80e-70cd-4ad6-b562-ca79cbe6e480" containerName="console"
Sep 18 19:47:00.627179 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.627154    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:00.629555 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.629532    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 18 19:47:00.629555 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.629544    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 18 19:47:00.629740 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.629551    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-l6bqs\""
Sep 18 19:47:00.629740 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.629532    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 18 19:47:00.632921 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.632890    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-v5vb4"]
Sep 18 19:47:00.677306 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.677276    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.710467 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.710447    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5da113bf-6f06-480e-9f1b-b48ea16f264f-data\") pod \"seaweedfs-64568bcd49-v5vb4\" (UID: \"5da113bf-6f06-480e-9f1b-b48ea16f264f\") " pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:00.710544 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.710496    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp52d\" (UniqueName: \"kubernetes.io/projected/5da113bf-6f06-480e-9f1b-b48ea16f264f-kube-api-access-vp52d\") pod \"seaweedfs-64568bcd49-v5vb4\" (UID: \"5da113bf-6f06-480e-9f1b-b48ea16f264f\") " pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:00.777351 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.777299    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.811508 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.811485    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5da113bf-6f06-480e-9f1b-b48ea16f264f-data\") pod \"seaweedfs-64568bcd49-v5vb4\" (UID: \"5da113bf-6f06-480e-9f1b-b48ea16f264f\") " pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:00.811601 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.811542    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp52d\" (UniqueName: \"kubernetes.io/projected/5da113bf-6f06-480e-9f1b-b48ea16f264f-kube-api-access-vp52d\") pod \"seaweedfs-64568bcd49-v5vb4\" (UID: \"5da113bf-6f06-480e-9f1b-b48ea16f264f\") " pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:00.811847 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.811829    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5da113bf-6f06-480e-9f1b-b48ea16f264f-data\") pod \"seaweedfs-64568bcd49-v5vb4\" (UID: \"5da113bf-6f06-480e-9f1b-b48ea16f264f\") " pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:00.819197 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.819172    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp52d\" (UniqueName: \"kubernetes.io/projected/5da113bf-6f06-480e-9f1b-b48ea16f264f-kube-api-access-vp52d\") pod \"seaweedfs-64568bcd49-v5vb4\" (UID: \"5da113bf-6f06-480e-9f1b-b48ea16f264f\") " pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:00.880452 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.880428    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:00.937571 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:00.937544    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:01.062512 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:01.062444    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:01.161308 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:01.161282    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:01.204740 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:01.204681    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-v5vb4"]
Sep 18 19:47:01.207479 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:47:01.207452    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da113bf_6f06_480e_9f1b_b48ea16f264f.slice/crio-95c11ac3955bc914fc329f44dea970fc005761344aad1d3865d32ffeec2dd3e5 WatchSource:0}: Error finding container 95c11ac3955bc914fc329f44dea970fc005761344aad1d3865d32ffeec2dd3e5: Status 404 returned error can't find the container with id 95c11ac3955bc914fc329f44dea970fc005761344aad1d3865d32ffeec2dd3e5
Sep 18 19:47:01.208686 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:01.208671    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:47:01.224743 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:01.224718    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-v5vb4" event={"ID":"5da113bf-6f06-480e-9f1b-b48ea16f264f","Type":"ContainerStarted","Data":"95c11ac3955bc914fc329f44dea970fc005761344aad1d3865d32ffeec2dd3e5"}
Sep 18 19:47:01.262808 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:01.262787    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:04.236590 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:04.236563    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-v5vb4" event={"ID":"5da113bf-6f06-480e-9f1b-b48ea16f264f","Type":"ContainerStarted","Data":"bfe9b8c375b62fc12b6a0a2629b040add7b8fad1b182d246b736638205a585f6"}
Sep 18 19:47:04.236899 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:04.236685    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:04.252309 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:04.252252    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-v5vb4" podStartSLOduration=1.3138075169999999 podStartE2EDuration="4.252237559s" totalImagesPullingTime="2.938430042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:47:04.251264335 +0000 UTC m=+515.213997073" watchObservedRunningTime="2026-09-18 19:47:04.252237559 +0000 UTC m=+515.214970297"
Sep 18 19:47:04.499994 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:04.499951    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:04.600937 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:04.600912    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:10.241588 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:10.241558    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-v5vb4"
Sep 18 19:47:56.259755 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:56.259707    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:47:56.310491 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:47:56.310458    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:17.428953 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.428920    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:17.452621 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.452594    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-vk5jt"]
Sep 18 19:48:17.455527 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.455512    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:17.457804 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.457781    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-kqt2b\""
Sep 18 19:48:17.457890 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.457835    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 18 19:48:17.471702 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.471669    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vk5jt"]
Sep 18 19:48:17.479682 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.479662    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-px9dz"]
Sep 18 19:48:17.482474 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.482457    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:17.484770 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.484750    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 18 19:48:17.484865 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.484786    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 18 19:48:17.484941 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.484923    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-pp4p9\""
Sep 18 19:48:17.494820 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.494799    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-px9dz"]
Sep 18 19:48:17.526487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.526456    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9123bf-09eb-4f12-94cf-f96097be22f6-tls-certs\") pod \"model-serving-api-7b954ffc95-vk5jt\" (UID: \"eb9123bf-09eb-4f12-94cf-f96097be22f6\") " pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:17.526631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.526501    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:17.526631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.526558    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb67v\" (UniqueName: \"kubernetes.io/projected/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-kube-api-access-vb67v\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:17.526631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.526618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-metrics-certs\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:17.526766 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.526645    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r55n\" (UniqueName: \"kubernetes.io/projected/eb9123bf-09eb-4f12-94cf-f96097be22f6-kube-api-access-2r55n\") pod \"model-serving-api-7b954ffc95-vk5jt\" (UID: \"eb9123bf-09eb-4f12-94cf-f96097be22f6\") " pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:17.627750 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.627711    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb67v\" (UniqueName: \"kubernetes.io/projected/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-kube-api-access-vb67v\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:17.627750 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.627761    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-metrics-certs\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:17.627951 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.627810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r55n\" (UniqueName: \"kubernetes.io/projected/eb9123bf-09eb-4f12-94cf-f96097be22f6-kube-api-access-2r55n\") pod \"model-serving-api-7b954ffc95-vk5jt\" (UID: \"eb9123bf-09eb-4f12-94cf-f96097be22f6\") " pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:17.627951 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:48:17.627849    2559 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 18 19:48:17.627951 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.627867    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9123bf-09eb-4f12-94cf-f96097be22f6-tls-certs\") pod \"model-serving-api-7b954ffc95-vk5jt\" (UID: \"eb9123bf-09eb-4f12-94cf-f96097be22f6\") " pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:17.627951 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:48:17.627896    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-metrics-certs podName:a2fdb58e-4daa-405c-94e4-da7c75a3f12d nodeName:}" failed. No retries permitted until 2026-09-18 19:48:18.127879569 +0000 UTC m=+589.090612285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-metrics-certs") pod "odh-model-controller-7b966d7984-px9dz" (UID: "a2fdb58e-4daa-405c-94e4-da7c75a3f12d") : secret "odh-model-controller-metrics-tls" not found
Sep 18 19:48:17.627951 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:48:17.627933    2559 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 18 19:48:17.628188 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:48:17.627985    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb9123bf-09eb-4f12-94cf-f96097be22f6-tls-certs podName:eb9123bf-09eb-4f12-94cf-f96097be22f6 nodeName:}" failed. No retries permitted until 2026-09-18 19:48:18.127972921 +0000 UTC m=+589.090705651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/eb9123bf-09eb-4f12-94cf-f96097be22f6-tls-certs") pod "model-serving-api-7b954ffc95-vk5jt" (UID: "eb9123bf-09eb-4f12-94cf-f96097be22f6") : secret "model-serving-api-tls" not found
Sep 18 19:48:17.628188 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.628157    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:17.628268 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:48:17.628239    2559 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 19:48:17.628308 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:48:17.628285    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert podName:a2fdb58e-4daa-405c-94e4-da7c75a3f12d nodeName:}" failed. No retries permitted until 2026-09-18 19:48:18.128275666 +0000 UTC m=+589.091008383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert") pod "odh-model-controller-7b966d7984-px9dz" (UID: "a2fdb58e-4daa-405c-94e4-da7c75a3f12d") : secret "odh-model-controller-webhook-cert" not found
Sep 18 19:48:17.635880 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.635853    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb67v\" (UniqueName: \"kubernetes.io/projected/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-kube-api-access-vb67v\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:17.636126 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:17.636109    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r55n\" (UniqueName: \"kubernetes.io/projected/eb9123bf-09eb-4f12-94cf-f96097be22f6-kube-api-access-2r55n\") pod \"model-serving-api-7b954ffc95-vk5jt\" (UID: \"eb9123bf-09eb-4f12-94cf-f96097be22f6\") " pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:18.132758 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:18.132718    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-metrics-certs\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:18.133006 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:18.132777    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9123bf-09eb-4f12-94cf-f96097be22f6-tls-certs\") pod \"model-serving-api-7b954ffc95-vk5jt\" (UID: \"eb9123bf-09eb-4f12-94cf-f96097be22f6\") " pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:18.133006 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:18.132801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:18.133006 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:48:18.132891    2559 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 19:48:18.133006 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:48:18.132940    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert podName:a2fdb58e-4daa-405c-94e4-da7c75a3f12d nodeName:}" failed. No retries permitted until 2026-09-18 19:48:19.132928075 +0000 UTC m=+590.095660790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert") pod "odh-model-controller-7b966d7984-px9dz" (UID: "a2fdb58e-4daa-405c-94e4-da7c75a3f12d") : secret "odh-model-controller-webhook-cert" not found
Sep 18 19:48:18.135211 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:18.135184    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-metrics-certs\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:18.135290 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:18.135281    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb9123bf-09eb-4f12-94cf-f96097be22f6-tls-certs\") pod \"model-serving-api-7b954ffc95-vk5jt\" (UID: \"eb9123bf-09eb-4f12-94cf-f96097be22f6\") " pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:18.365062 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:18.365030    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:18.643497 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:18.643464    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-vk5jt"]
Sep 18 19:48:18.646543 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:48:18.646515    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9123bf_09eb_4f12_94cf_f96097be22f6.slice/crio-fa9aa7310875d77e58cf86fedb8db3582cee6847046e8f0d6f9f97d2c039f7bb WatchSource:0}: Error finding container fa9aa7310875d77e58cf86fedb8db3582cee6847046e8f0d6f9f97d2c039f7bb: Status 404 returned error can't find the container with id fa9aa7310875d77e58cf86fedb8db3582cee6847046e8f0d6f9f97d2c039f7bb
Sep 18 19:48:18.729808 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:18.729773    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:19.142352 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:19.142243    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:19.144693 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:19.144670    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2fdb58e-4daa-405c-94e4-da7c75a3f12d-cert\") pod \"odh-model-controller-7b966d7984-px9dz\" (UID: \"a2fdb58e-4daa-405c-94e4-da7c75a3f12d\") " pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:19.291763 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:19.291724    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:19.493973 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:19.493923    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vk5jt" event={"ID":"eb9123bf-09eb-4f12-94cf-f96097be22f6","Type":"ContainerStarted","Data":"fa9aa7310875d77e58cf86fedb8db3582cee6847046e8f0d6f9f97d2c039f7bb"}
Sep 18 19:48:19.575613 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:19.575528    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-px9dz"]
Sep 18 19:48:19.579478 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:48:19.579446    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fdb58e_4daa_405c_94e4_da7c75a3f12d.slice/crio-14ebcf6a34b0a7cb7b1feb73a1d41cb4242ef28960b946636c7252648a98c6ef WatchSource:0}: Error finding container 14ebcf6a34b0a7cb7b1feb73a1d41cb4242ef28960b946636c7252648a98c6ef: Status 404 returned error can't find the container with id 14ebcf6a34b0a7cb7b1feb73a1d41cb4242ef28960b946636c7252648a98c6ef
Sep 18 19:48:20.499709 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:20.499668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-px9dz" event={"ID":"a2fdb58e-4daa-405c-94e4-da7c75a3f12d","Type":"ContainerStarted","Data":"14ebcf6a34b0a7cb7b1feb73a1d41cb4242ef28960b946636c7252648a98c6ef"}
Sep 18 19:48:21.505254 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:21.505224    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-vk5jt" event={"ID":"eb9123bf-09eb-4f12-94cf-f96097be22f6","Type":"ContainerStarted","Data":"424127f316788b542251891063a582d4670c55a22508059ef360dd2a75977cab"}
Sep 18 19:48:21.505622 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:21.505359    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:21.524272 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:21.522451    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-vk5jt" podStartSLOduration=2.188511956 podStartE2EDuration="4.522429546s" totalImagesPullingTime="2.33391759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:21.519728061 +0000 UTC m=+592.482460800" watchObservedRunningTime="2026-09-18 19:48:21.522429546 +0000 UTC m=+592.485162284"
Sep 18 19:48:23.515146 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:23.515112    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-px9dz" event={"ID":"a2fdb58e-4daa-405c-94e4-da7c75a3f12d","Type":"ContainerStarted","Data":"780ad3d802ab49b5600341997246e8938514560d817a069fc5725d4ec7cc5d10"}
Sep 18 19:48:23.515527 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:23.515247    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:23.531754 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:23.531682    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-px9dz" podStartSLOduration=3.467421475 podStartE2EDuration="6.531664255s" totalImagesPullingTime="3.06424278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:48:23.530822601 +0000 UTC m=+594.493555339" watchObservedRunningTime="2026-09-18 19:48:23.531664255 +0000 UTC m=+594.494396993"
Sep 18 19:48:31.353975 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:31.353929    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:48:31.354524 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:31.353940    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:48:32.514185 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:32.514155    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-vk5jt"
Sep 18 19:48:33.580549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:33.580494    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:33.655151 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:33.655119    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:33.738868 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:33.738825    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:34.307841 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:34.307807    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:34.410747 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:34.410717    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:34.521590 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:34.521562    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-px9dz"
Sep 18 19:48:44.544107 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:44.544026    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:44.645150 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:44.645117    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:48:44.705461 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:48:44.705433    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:54.508735 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:49:54.508703    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:49:54.621991 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:49:54.621934    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:05.939573 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:05.939537    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:05.994612 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:05.994577    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"]
Sep 18 19:50:05.997843 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:05.997828    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.000249 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.000221    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 18 19:50:06.000249 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.000223    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:50:06.000428 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.000256    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 18 19:50:06.000428 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.000286    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:50:06.000428 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.000403    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:50:06.000428 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.000423    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:50:06.000947 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.000930    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\""
Sep 18 19:50:06.006060 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.006035    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"]
Sep 18 19:50:06.075929 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.075888    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:06.102084 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.102053    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"]
Sep 18 19:50:06.105133 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.105113    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.107135 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.107120    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 18 19:50:06.114865 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.114847    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"]
Sep 18 19:50:06.164114 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.164090    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.164251 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.164122    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8pps\" (UniqueName: \"kubernetes.io/projected/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-api-access-j8pps\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.164251 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.164158    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.164251 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.164185    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.164251 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.164234    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.203432 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.203406    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:06.264695 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.264675    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.264809 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.264707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.264809 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.264730    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.264809 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.264747    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.264809 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.264784    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.265003 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:50:06.264850    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 18 19:50:06.265003 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:50:06.264926    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls podName:0663e6db-fe95-4012-9d4f-e0e40f8e6b65 nodeName:}" failed. No retries permitted until 2026-09-18 19:50:06.764905729 +0000 UTC m=+697.727638453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls") pod "isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" (UID: "0663e6db-fe95-4012-9d4f-e0e40f8e6b65") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 18 19:50:06.265003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.264984    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.265157 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.265018    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.265157 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.265053    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8pps\" (UniqueName: \"kubernetes.io/projected/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-api-access-j8pps\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.265157 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.265102    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.265157 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.265128    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dq6d\" (UniqueName: \"kubernetes.io/projected/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-api-access-2dq6d\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.265382 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.265362    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.265532 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.265513    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.265727 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.265709    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.287506 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.287479    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8pps\" (UniqueName: \"kubernetes.io/projected/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-api-access-j8pps\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.366089 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.366059    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dq6d\" (UniqueName: \"kubernetes.io/projected/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-api-access-2dq6d\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.366221 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.366118    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.366221 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.366161    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.366221 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.366189    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.366375 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.366244    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.366671 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.366643    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.366945 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.366925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.367019 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.366925    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.368680 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.368661    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.374082 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.374062    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dq6d\" (UniqueName: \"kubernetes.io/projected/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-api-access-2dq6d\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-5mhzk\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.381370 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.381351    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:06.415443 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.415409    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:06.447881 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.447857    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:06.548854 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.548820    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:06.689365 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.689339    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"]
Sep 18 19:50:06.692558 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:50:06.692531    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7c9b7d_6459_4084_9ed4_10acd85aa3e2.slice/crio-6cf30858346d5b65c73c21a256df65665b24b9b76a7a7fd3a36fc3f17a51ec59 WatchSource:0}: Error finding container 6cf30858346d5b65c73c21a256df65665b24b9b76a7a7fd3a36fc3f17a51ec59: Status 404 returned error can't find the container with id 6cf30858346d5b65c73c21a256df65665b24b9b76a7a7fd3a36fc3f17a51ec59
Sep 18 19:50:06.769753 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.769693    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.772020 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.772000    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6946b55967-dt4pf\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:06.875548 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.875521    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" event={"ID":"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2","Type":"ContainerStarted","Data":"6cf30858346d5b65c73c21a256df65665b24b9b76a7a7fd3a36fc3f17a51ec59"}
Sep 18 19:50:06.908220 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:06.908198    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:07.176035 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:07.175825    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"]
Sep 18 19:50:07.178547 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:50:07.178519    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0663e6db_fe95_4012_9d4f_e0e40f8e6b65.slice/crio-1ed05bde99bec80050133a0cc9af2d69cff958da223557f2c493db83b12c1da4 WatchSource:0}: Error finding container 1ed05bde99bec80050133a0cc9af2d69cff958da223557f2c493db83b12c1da4: Status 404 returned error can't find the container with id 1ed05bde99bec80050133a0cc9af2d69cff958da223557f2c493db83b12c1da4
Sep 18 19:50:07.881811 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:07.881650    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" event={"ID":"0663e6db-fe95-4012-9d4f-e0e40f8e6b65","Type":"ContainerStarted","Data":"1ed05bde99bec80050133a0cc9af2d69cff958da223557f2c493db83b12c1da4"}
Sep 18 19:50:10.897648 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:10.897611    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" event={"ID":"0663e6db-fe95-4012-9d4f-e0e40f8e6b65","Type":"ContainerStarted","Data":"2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff"}
Sep 18 19:50:10.899075 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:10.899047    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" event={"ID":"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2","Type":"ContainerStarted","Data":"c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e"}
Sep 18 19:50:13.911226 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:13.911198    2559 generic.go:356] "Generic (PLEG): container finished" podID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerID="c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e" exitCode=0
Sep 18 19:50:13.911612 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:13.911279    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" event={"ID":"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2","Type":"ContainerDied","Data":"c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e"}
Sep 18 19:50:13.912957 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:13.912911    2559 generic.go:356] "Generic (PLEG): container finished" podID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerID="2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff" exitCode=0
Sep 18 19:50:13.913041 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:13.912977    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" event={"ID":"0663e6db-fe95-4012-9d4f-e0e40f8e6b65","Type":"ContainerDied","Data":"2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff"}
Sep 18 19:50:31.021390 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:31.021168    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" event={"ID":"0663e6db-fe95-4012-9d4f-e0e40f8e6b65","Type":"ContainerStarted","Data":"b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7"}
Sep 18 19:50:31.029899 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:31.029722    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" event={"ID":"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2","Type":"ContainerStarted","Data":"3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f"}
Sep 18 19:50:33.040104 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:33.040065    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" event={"ID":"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2","Type":"ContainerStarted","Data":"642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522"}
Sep 18 19:50:34.045432 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:34.045388    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" event={"ID":"0663e6db-fe95-4012-9d4f-e0e40f8e6b65","Type":"ContainerStarted","Data":"2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05"}
Sep 18 19:50:34.045924 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:34.045563    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:34.045924 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:34.045627    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:34.046946 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:34.046919    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:50:34.101623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:34.101568    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podStartSLOduration=2.116966166 podStartE2EDuration="28.101551989s" totalImagesPullingTime="22.37595127s" totalInitContainerRuntime="3.608634553s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:34.099701843 +0000 UTC m=+725.062434597" watchObservedRunningTime="2026-09-18 19:50:34.101551989 +0000 UTC m=+725.064284726"
Sep 18 19:50:34.130475 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:34.130421    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podStartSLOduration=3.452237559 podStartE2EDuration="29.130404144s" totalImagesPullingTime="22.06709864s" totalInitContainerRuntime="3.611067945s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:50:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:50:34.12542155 +0000 UTC m=+725.088154289" watchObservedRunningTime="2026-09-18 19:50:34.130404144 +0000 UTC m=+725.093136882"
Sep 18 19:50:35.050062 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:35.050019    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:50:35.050487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:35.050067    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:35.050487 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:35.050124    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:35.051143 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:35.051121    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:50:36.053620 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:36.053577    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:50:40.054152 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:40.054114    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:50:40.054642 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:40.054612    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:50:41.058465 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:41.058436    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:50:41.059027 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:41.058997    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:50:46.389797 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:46.389756    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:46.490314 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:46.490281    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:50:50.054928 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:50.054880    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:50:51.058876 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:50:51.058841    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:51:00.055172 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:00.055131    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:51:01.059618 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:01.059575    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:51:10.055101 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:10.055061    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:51:11.059046 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:11.059008    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:51:20.055016 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:20.054957    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:51:21.058838 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:21.058801    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:51:26.025212 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:26.025178    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:30.055303 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:30.055260    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:51:31.059426 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:31.059393    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:51:40.055139 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:40.055095    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.18:8080: connect: connection refused"
Sep 18 19:51:41.059047 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:41.059004    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.17:8080: connect: connection refused"
Sep 18 19:51:50.055477 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:50.055395    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:51:51.059595 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:51.059568    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:51:51.177556 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:51.177522    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"]
Sep 18 19:51:51.177772 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:51.177728    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container" containerID="cri-o://3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f" gracePeriod=30
Sep 18 19:51:51.177845 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:51.177770    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kube-rbac-proxy" containerID="cri-o://642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522" gracePeriod=30
Sep 18 19:51:51.338578 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:51.338501    2559 generic.go:356] "Generic (PLEG): container finished" podID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerID="642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522" exitCode=2
Sep 18 19:51:51.338578 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:51.338561    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" event={"ID":"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2","Type":"ContainerDied","Data":"642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522"}
Sep 18 19:51:55.050357 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:55.050303    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.18:8643/healthz\": dial tcp 10.134.0.18:8643: connect: connection refused"
Sep 18 19:51:55.878207 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:55.878174    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"]
Sep 18 19:51:55.878497 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:55.878472    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container" containerID="cri-o://b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7" gracePeriod=30
Sep 18 19:51:55.878745 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:55.878553    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kube-rbac-proxy" containerID="cri-o://2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05" gracePeriod=30
Sep 18 19:51:55.908469 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:55.908446    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:51:56.000685 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.000660    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:51:56.014022 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.013948    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"]
Sep 18 19:51:56.014236 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.014222    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kube-rbac-proxy"
Sep 18 19:51:56.014282 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.014237    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kube-rbac-proxy"
Sep 18 19:51:56.014282 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.014253    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container"
Sep 18 19:51:56.014282 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.014259    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container"
Sep 18 19:51:56.014282 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.014266    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="storage-initializer"
Sep 18 19:51:56.014282 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.014272    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="storage-initializer"
Sep 18 19:51:56.014430 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.014314    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kserve-container"
Sep 18 19:51:56.014430 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.014322    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerName="kube-rbac-proxy"
Sep 18 19:51:56.016020 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.016006    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.026431 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.026410    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"]
Sep 18 19:51:56.053558 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.053525    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.17:8643/healthz\": dial tcp 10.134.0.17:8643: connect: connection refused"
Sep 18 19:51:56.160519 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.160481    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-proxy-tls\") pod \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") "
Sep 18 19:51:56.160519 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.160520    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dq6d\" (UniqueName: \"kubernetes.io/projected/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-api-access-2dq6d\") pod \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") "
Sep 18 19:51:56.160721 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.160553    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-cabundle-cert\") pod \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") "
Sep 18 19:51:56.160721 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.160675    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kserve-provision-location\") pod \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") "
Sep 18 19:51:56.160812 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.160792    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-rbac-proxy-sar-config\") pod \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\" (UID: \"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2\") "
Sep 18 19:51:56.160930 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.160903    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/825423dc-e875-4452-837c-aa529baafdb8-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.161035 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.160936    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" (UID: "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:56.161035 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.160932    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" (UID: "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:51:56.161142 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.161119    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlztc\" (UniqueName: \"kubernetes.io/projected/825423dc-e875-4452-837c-aa529baafdb8-kube-api-access-wlztc\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.161185 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.161140    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" (UID: "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:51:56.161227 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.161209    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.161266 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.161244    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.161313 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.161277    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/825423dc-e875-4452-837c-aa529baafdb8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.161352 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.161331    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:51:56.161352 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.161341    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:51:56.161352 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.161350    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:51:56.162817 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.162798    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" (UID: "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:51:56.162863 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.162851    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-api-access-2dq6d" (OuterVolumeSpecName: "kube-api-access-2dq6d") pod "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" (UID: "0b7c9b7d-6459-4084-9ed4-10acd85aa3e2"). InnerVolumeSpecName "kube-api-access-2dq6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:51:56.261842 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.261810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/825423dc-e875-4452-837c-aa529baafdb8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.262001 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.261854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/825423dc-e875-4452-837c-aa529baafdb8-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.262001 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.261888    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlztc\" (UniqueName: \"kubernetes.io/projected/825423dc-e875-4452-837c-aa529baafdb8-kube-api-access-wlztc\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.262001 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.261915    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.262001 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.261935    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.262001 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.261989    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:51:56.262001 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.262002    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dq6d\" (UniqueName: \"kubernetes.io/projected/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2-kube-api-access-2dq6d\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:51:56.262330 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.262202    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/825423dc-e875-4452-837c-aa529baafdb8-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.262532 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.262511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.262638 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.262618    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-cabundle-cert\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.264345 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.264304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/825423dc-e875-4452-837c-aa529baafdb8-proxy-tls\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.270089 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.270072    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlztc\" (UniqueName: \"kubernetes.io/projected/825423dc-e875-4452-837c-aa529baafdb8-kube-api-access-wlztc\") pod \"isvc-canary-rollback-predictor-585bd9bcc8-9fltn\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.326874 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.326835    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:51:56.359560 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.359517    2559 generic.go:356] "Generic (PLEG): container finished" podID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" containerID="3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f" exitCode=0
Sep 18 19:51:56.359710 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.359603    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"
Sep 18 19:51:56.359771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.359595    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" event={"ID":"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2","Type":"ContainerDied","Data":"3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f"}
Sep 18 19:51:56.359816 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.359776    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk" event={"ID":"0b7c9b7d-6459-4084-9ed4-10acd85aa3e2","Type":"ContainerDied","Data":"6cf30858346d5b65c73c21a256df65665b24b9b76a7a7fd3a36fc3f17a51ec59"}
Sep 18 19:51:56.359857 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.359815    2559 scope.go:117] "RemoveContainer" containerID="642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522"
Sep 18 19:51:56.362056 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.361946    2559 generic.go:356] "Generic (PLEG): container finished" podID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerID="2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05" exitCode=2
Sep 18 19:51:56.362169 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.361992    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" event={"ID":"0663e6db-fe95-4012-9d4f-e0e40f8e6b65","Type":"ContainerDied","Data":"2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05"}
Sep 18 19:51:56.368747 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.368730    2559 scope.go:117] "RemoveContainer" containerID="3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f"
Sep 18 19:51:56.382710 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.382269    2559 scope.go:117] "RemoveContainer" containerID="c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e"
Sep 18 19:51:56.394299 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.394276    2559 scope.go:117] "RemoveContainer" containerID="642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522"
Sep 18 19:51:56.394763 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:51:56.394735    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522\": container with ID starting with 642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522 not found: ID does not exist" containerID="642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522"
Sep 18 19:51:56.394981 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.394776    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522"} err="failed to get container status \"642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522\": rpc error: code = NotFound desc = could not find container \"642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522\": container with ID starting with 642034d16150c49d1cba57554c5fcafdae6ae558bb77b46d90134fcfe65a1522 not found: ID does not exist"
Sep 18 19:51:56.394981 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.394816    2559 scope.go:117] "RemoveContainer" containerID="3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f"
Sep 18 19:51:56.395396 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:51:56.395372    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f\": container with ID starting with 3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f not found: ID does not exist" containerID="3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f"
Sep 18 19:51:56.395396 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.395403    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f"} err="failed to get container status \"3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f\": rpc error: code = NotFound desc = could not find container \"3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f\": container with ID starting with 3b4dd231727601e90b7ff1d126ed6582d78529bae45fb2819abe7ef097c52d6f not found: ID does not exist"
Sep 18 19:51:56.395396 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.395422    2559 scope.go:117] "RemoveContainer" containerID="c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e"
Sep 18 19:51:56.395733 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.395685    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"]
Sep 18 19:51:56.395853 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:51:56.395835    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e\": container with ID starting with c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e not found: ID does not exist" containerID="c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e"
Sep 18 19:51:56.395909 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.395858    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e"} err="failed to get container status \"c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e\": rpc error: code = NotFound desc = could not find container \"c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e\": container with ID starting with c8106adf9af0bf5471a0dee2aa503c9afb18c7e259d9aea9becc9c481564c31e not found: ID does not exist"
Sep 18 19:51:56.403052 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.403031    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-5mhzk"]
Sep 18 19:51:56.601307 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:56.601285    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"]
Sep 18 19:51:56.603388 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:51:56.603359    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825423dc_e875_4452_837c_aa529baafdb8.slice/crio-9aa8ee7452263dbf3c1450ce4ecd0877aad7f50d2e5f46fdcf011f8e72957a8b WatchSource:0}: Error finding container 9aa8ee7452263dbf3c1450ce4ecd0877aad7f50d2e5f46fdcf011f8e72957a8b: Status 404 returned error can't find the container with id 9aa8ee7452263dbf3c1450ce4ecd0877aad7f50d2e5f46fdcf011f8e72957a8b
Sep 18 19:51:57.368570 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:57.368534    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" event={"ID":"825423dc-e875-4452-837c-aa529baafdb8","Type":"ContainerStarted","Data":"26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd"}
Sep 18 19:51:57.368948 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:57.368591    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" event={"ID":"825423dc-e875-4452-837c-aa529baafdb8","Type":"ContainerStarted","Data":"9aa8ee7452263dbf3c1450ce4ecd0877aad7f50d2e5f46fdcf011f8e72957a8b"}
Sep 18 19:51:57.470262 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:51:57.470230    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b7c9b7d-6459-4084-9ed4-10acd85aa3e2" path="/var/lib/kubelet/pods/0b7c9b7d-6459-4084-9ed4-10acd85aa3e2/volumes"
Sep 18 19:52:00.711917 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.711893    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:52:00.796655 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.796619    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-cabundle-cert\") pod \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") "
Sep 18 19:52:00.796655 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.796664    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-rbac-proxy-sar-config\") pod \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") "
Sep 18 19:52:00.796910 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.796879    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8pps\" (UniqueName: \"kubernetes.io/projected/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-api-access-j8pps\") pod \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") "
Sep 18 19:52:00.797078 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.796940    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls\") pod \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") "
Sep 18 19:52:00.797078 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.797013    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0663e6db-fe95-4012-9d4f-e0e40f8e6b65" (UID: "0663e6db-fe95-4012-9d4f-e0e40f8e6b65"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:00.797078 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.797038    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kserve-provision-location\") pod \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\" (UID: \"0663e6db-fe95-4012-9d4f-e0e40f8e6b65\") "
Sep 18 19:52:00.797078 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.797042    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0663e6db-fe95-4012-9d4f-e0e40f8e6b65" (UID: "0663e6db-fe95-4012-9d4f-e0e40f8e6b65"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:52:00.797281 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.797200    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.797281 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.797211    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.797393 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.797362    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0663e6db-fe95-4012-9d4f-e0e40f8e6b65" (UID: "0663e6db-fe95-4012-9d4f-e0e40f8e6b65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:52:00.798987 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.798954    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-api-access-j8pps" (OuterVolumeSpecName: "kube-api-access-j8pps") pod "0663e6db-fe95-4012-9d4f-e0e40f8e6b65" (UID: "0663e6db-fe95-4012-9d4f-e0e40f8e6b65"). InnerVolumeSpecName "kube-api-access-j8pps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:52:00.799054 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.799022    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0663e6db-fe95-4012-9d4f-e0e40f8e6b65" (UID: "0663e6db-fe95-4012-9d4f-e0e40f8e6b65"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:52:00.898519 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.898488    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.898519 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.898516    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8pps\" (UniqueName: \"kubernetes.io/projected/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-kube-api-access-j8pps\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.898519 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.898526    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0663e6db-fe95-4012-9d4f-e0e40f8e6b65-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:52:00.987348 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:00.987317    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:01.085149 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.085111    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:01.113687 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.113656    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"]
Sep 18 19:52:01.113929 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.113917    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container"
Sep 18 19:52:01.113986 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.113930    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container"
Sep 18 19:52:01.113986 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.113952    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kube-rbac-proxy"
Sep 18 19:52:01.113986 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.113957    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kube-rbac-proxy"
Sep 18 19:52:01.114153 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.113985    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="storage-initializer"
Sep 18 19:52:01.114153 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.113995    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="storage-initializer"
Sep 18 19:52:01.114153 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.114036    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kserve-container"
Sep 18 19:52:01.114153 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.114042    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerName="kube-rbac-proxy"
Sep 18 19:52:01.116817 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.116799    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.119033 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.119011    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-2f453-predictor-serving-cert\""
Sep 18 19:52:01.119033 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.119023    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-2f453-kube-rbac-proxy-sar-config\""
Sep 18 19:52:01.128798 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.128772    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"]
Sep 18 19:52:01.185319 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.185290    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:01.201446 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.201412    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.201579 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.201453    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292307d8-3fb5-4b4a-9531-fa2f622588c4-proxy-tls\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.201579 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.201481    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292307d8-3fb5-4b4a-9531-fa2f622588c4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.201579 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.201547    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.201703 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.201581    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbsm\" (UniqueName: \"kubernetes.io/projected/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-api-access-jkbsm\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.211033 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.211003    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"]
Sep 18 19:52:01.213828 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.213814    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.215871 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.215856    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-2f453-kube-rbac-proxy-sar-config\""
Sep 18 19:52:01.215955 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.215911    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-2f453-predictor-serving-cert\""
Sep 18 19:52:01.224842 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.224823    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"]
Sep 18 19:52:01.302718 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.302632    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.302718 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.302670    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.302718 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.302703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.303011 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.302725    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.303011 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.302742    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292307d8-3fb5-4b4a-9531-fa2f622588c4-proxy-tls\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.303011 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.302758    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292307d8-3fb5-4b4a-9531-fa2f622588c4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.303011 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.302783    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.303011 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.302801    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdkd\" (UniqueName: \"kubernetes.io/projected/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-api-access-bwdkd\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.303269 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.303023    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbsm\" (UniqueName: \"kubernetes.io/projected/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-api-access-jkbsm\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.303269 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.303065    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.303269 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.303227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292307d8-3fb5-4b4a-9531-fa2f622588c4-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.303552 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.303529    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-cabundle-cert\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.303602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.303586    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.305249 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.305227    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292307d8-3fb5-4b4a-9531-fa2f622588c4-proxy-tls\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.310613 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.310591    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbsm\" (UniqueName: \"kubernetes.io/projected/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-api-access-jkbsm\") pod \"isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.312862 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.312842    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:01.385695 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.385655    2559 generic.go:356] "Generic (PLEG): container finished" podID="825423dc-e875-4452-837c-aa529baafdb8" containerID="26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd" exitCode=0
Sep 18 19:52:01.385862 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.385729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" event={"ID":"825423dc-e875-4452-837c-aa529baafdb8","Type":"ContainerDied","Data":"26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd"}
Sep 18 19:52:01.387718 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.387694    2559 generic.go:356] "Generic (PLEG): container finished" podID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" containerID="b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7" exitCode=0
Sep 18 19:52:01.387824 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.387755    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" event={"ID":"0663e6db-fe95-4012-9d4f-e0e40f8e6b65","Type":"ContainerDied","Data":"b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7"}
Sep 18 19:52:01.387824 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.387763    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"
Sep 18 19:52:01.387906 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.387808    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf" event={"ID":"0663e6db-fe95-4012-9d4f-e0e40f8e6b65","Type":"ContainerDied","Data":"1ed05bde99bec80050133a0cc9af2d69cff958da223557f2c493db83b12c1da4"}
Sep 18 19:52:01.387906 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.387881    2559 scope.go:117] "RemoveContainer" containerID="2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05"
Sep 18 19:52:01.398272 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.398253    2559 scope.go:117] "RemoveContainer" containerID="b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7"
Sep 18 19:52:01.404066 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.404044    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdkd\" (UniqueName: \"kubernetes.io/projected/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-api-access-bwdkd\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.404169 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.404092    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.404169 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.404143    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.404276 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.404171    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.404276 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.404219    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.404375 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:52:01.404339    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-serving-cert: secret "isvc-xgboost-graph-raw-2f453-predictor-serving-cert" not found
Sep 18 19:52:01.404425 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:52:01.404405    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls podName:a564cac9-6b13-4726-8f8f-ebd8f7af0b57 nodeName:}" failed. No retries permitted until 2026-09-18 19:52:01.904385769 +0000 UTC m=+812.867118508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls") pod "isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" (UID: "a564cac9-6b13-4726-8f8f-ebd8f7af0b57") : secret "isvc-xgboost-graph-raw-2f453-predictor-serving-cert" not found
Sep 18 19:52:01.404551 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.404526    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.404926 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.404901    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-cabundle-cert\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.405066 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.405049    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.411873 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.411850    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdkd\" (UniqueName: \"kubernetes.io/projected/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-api-access-bwdkd\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.414657 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.414634    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"]
Sep 18 19:52:01.421095 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.421071    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6946b55967-dt4pf"]
Sep 18 19:52:01.422566 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.422531    2559 scope.go:117] "RemoveContainer" containerID="2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff"
Sep 18 19:52:01.426697 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.426674    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:01.436409 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.436386    2559 scope.go:117] "RemoveContainer" containerID="2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05"
Sep 18 19:52:01.437286 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:52:01.437214    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05\": container with ID starting with 2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05 not found: ID does not exist" containerID="2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05"
Sep 18 19:52:01.437286 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.437249    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05"} err="failed to get container status \"2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05\": rpc error: code = NotFound desc = could not find container \"2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05\": container with ID starting with 2c141efd6016c3bd0458724de32d38b4600145d5c35e68e88ecd0350f90a2f05 not found: ID does not exist"
Sep 18 19:52:01.437286 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.437276    2559 scope.go:117] "RemoveContainer" containerID="b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7"
Sep 18 19:52:01.437562 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:52:01.437538    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7\": container with ID starting with b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7 not found: ID does not exist" containerID="b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7"
Sep 18 19:52:01.437599 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.437573    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7"} err="failed to get container status \"b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7\": rpc error: code = NotFound desc = could not find container \"b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7\": container with ID starting with b1912ff690af0c7ca46a1c5673d2b594b6e4ebbcba628467c774c70753959bd7 not found: ID does not exist"
Sep 18 19:52:01.437638 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.437597    2559 scope.go:117] "RemoveContainer" containerID="2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff"
Sep 18 19:52:01.437880 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:52:01.437861    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff\": container with ID starting with 2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff not found: ID does not exist" containerID="2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff"
Sep 18 19:52:01.437925 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.437886    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff"} err="failed to get container status \"2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff\": rpc error: code = NotFound desc = could not find container \"2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff\": container with ID starting with 2a241087dabab665196155471a914a5f5cc04c68a6178869b0d1492a47dd60ff not found: ID does not exist"
Sep 18 19:52:01.470572 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.470545    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0663e6db-fe95-4012-9d4f-e0e40f8e6b65" path="/var/lib/kubelet/pods/0663e6db-fe95-4012-9d4f-e0e40f8e6b65/volumes"
Sep 18 19:52:01.582280 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.582187    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:52:01.707602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.707568    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"]
Sep 18 19:52:01.710464 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:52:01.710437    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292307d8_3fb5_4b4a_9531_fa2f622588c4.slice/crio-06df14b77ee766ab5a6b03a0c8762a89159263493de9c23a174df43eb9274d24 WatchSource:0}: Error finding container 06df14b77ee766ab5a6b03a0c8762a89159263493de9c23a174df43eb9274d24: Status 404 returned error can't find the container with id 06df14b77ee766ab5a6b03a0c8762a89159263493de9c23a174df43eb9274d24
Sep 18 19:52:01.909495 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.909397    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:01.911667 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:01.911640    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls\") pod \"isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:02.123298 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.123256    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:02.394392 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.394360    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" event={"ID":"825423dc-e875-4452-837c-aa529baafdb8","Type":"ContainerStarted","Data":"3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921"}
Sep 18 19:52:02.394565 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.394434    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" event={"ID":"825423dc-e875-4452-837c-aa529baafdb8","Type":"ContainerStarted","Data":"f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63"}
Sep 18 19:52:02.394890 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.394789    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:52:02.394890 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.394849    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:52:02.395343 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.395298    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"]
Sep 18 19:52:02.396066 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.396036    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:52:02.397046 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.397026    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" event={"ID":"292307d8-3fb5-4b4a-9531-fa2f622588c4","Type":"ContainerStarted","Data":"90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420"}
Sep 18 19:52:02.397150 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.397071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" event={"ID":"292307d8-3fb5-4b4a-9531-fa2f622588c4","Type":"ContainerStarted","Data":"06df14b77ee766ab5a6b03a0c8762a89159263493de9c23a174df43eb9274d24"}
Sep 18 19:52:02.398151 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:52:02.398130    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda564cac9_6b13_4726_8f8f_ebd8f7af0b57.slice/crio-3c84ecbbd17ccf7dfc4be97a082f0c2703cd8332aa6ea73bee852f3f3b50c9e1 WatchSource:0}: Error finding container 3c84ecbbd17ccf7dfc4be97a082f0c2703cd8332aa6ea73bee852f3f3b50c9e1: Status 404 returned error can't find the container with id 3c84ecbbd17ccf7dfc4be97a082f0c2703cd8332aa6ea73bee852f3f3b50c9e1
Sep 18 19:52:02.453763 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:02.453726    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podStartSLOduration=3.706879211 podStartE2EDuration="7.453709585s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.746830374s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:02.418385782 +0000 UTC m=+813.381118520" watchObservedRunningTime="2026-09-18 19:52:02.453709585 +0000 UTC m=+813.416442362"
Sep 18 19:52:03.402927 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:03.402889    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" event={"ID":"a564cac9-6b13-4726-8f8f-ebd8f7af0b57","Type":"ContainerStarted","Data":"a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15"}
Sep 18 19:52:03.403391 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:03.402950    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" event={"ID":"a564cac9-6b13-4726-8f8f-ebd8f7af0b57","Type":"ContainerStarted","Data":"3c84ecbbd17ccf7dfc4be97a082f0c2703cd8332aa6ea73bee852f3f3b50c9e1"}
Sep 18 19:52:03.403391 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:03.403175    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:52:06.415919 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:06.415891    2559 generic.go:356] "Generic (PLEG): container finished" podID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerID="90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420" exitCode=0
Sep 18 19:52:06.416259 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:06.415975    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" event={"ID":"292307d8-3fb5-4b4a-9531-fa2f622588c4","Type":"ContainerDied","Data":"90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420"}
Sep 18 19:52:06.417581 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:06.417560    2559 generic.go:356] "Generic (PLEG): container finished" podID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerID="a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15" exitCode=0
Sep 18 19:52:06.417660 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:06.417607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" event={"ID":"a564cac9-6b13-4726-8f8f-ebd8f7af0b57","Type":"ContainerDied","Data":"a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15"}
Sep 18 19:52:06.418821 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:06.418807    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 19:52:07.425833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:07.425792    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" event={"ID":"292307d8-3fb5-4b4a-9531-fa2f622588c4","Type":"ContainerStarted","Data":"12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51"}
Sep 18 19:52:07.426317 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:07.425872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" event={"ID":"292307d8-3fb5-4b4a-9531-fa2f622588c4","Type":"ContainerStarted","Data":"a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418"}
Sep 18 19:52:07.426317 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:07.426214    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:07.426429 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:07.426364    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:07.427935 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:07.427898    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:52:07.444719 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:07.444657    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podStartSLOduration=2.786898941 podStartE2EDuration="6.444641782s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.657742841s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:07.444355757 +0000 UTC m=+818.407088497" watchObservedRunningTime="2026-09-18 19:52:07.444641782 +0000 UTC m=+818.407374521"
Sep 18 19:52:08.409222 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:08.409192    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:52:08.409977 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:08.409771    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:52:08.431567 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:08.431534    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:52:13.436861 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:13.436831    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:52:13.437538 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:13.437503    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:52:18.410062 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:18.410015    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:52:23.437833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:23.437797    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:52:25.507595 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:25.507564    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" event={"ID":"a564cac9-6b13-4726-8f8f-ebd8f7af0b57","Type":"ContainerStarted","Data":"c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f"}
Sep 18 19:52:26.513796 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:26.513764    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" event={"ID":"a564cac9-6b13-4726-8f8f-ebd8f7af0b57","Type":"ContainerStarted","Data":"0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8"}
Sep 18 19:52:26.514241 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:26.513822    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:26.514241 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:26.513972    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:26.515331 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:26.515307    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:52:26.537314 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:26.537274    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podStartSLOduration=2.669043919 podStartE2EDuration="25.537262384s" totalImagesPullingTime="18.928609431s" totalInitContainerRuntime="3.939609034s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:52:26.535773935 +0000 UTC m=+837.498506672" watchObservedRunningTime="2026-09-18 19:52:26.537262384 +0000 UTC m=+837.499995123"
Sep 18 19:52:27.517751 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:27.517714    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:52:28.409836 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:28.409792    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:52:32.522404 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:32.522373    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:52:32.522901 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:32.522871    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:52:33.437601 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:33.437569    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:52:38.410248 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:38.410213    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:52:42.523692 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:42.523651    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:52:43.437566 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:43.437527    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:52:48.410561 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:48.410519    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:52:52.523144 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:52.523106    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:52:53.438067 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:53.438024    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:52:58.410448 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:52:58.410403    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:53:02.523181 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:02.523142    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:53:03.438261 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:03.438223    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:53:08.409897 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:08.409853    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:53:12.523380 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:12.523291    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:53:13.437884 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:13.437843    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:53:18.410151 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:18.410118    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:53:22.522936 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:22.522898    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:53:23.438151 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:23.438122    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:53:26.074519 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:26.074487    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"]
Sep 18 19:53:26.074979 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:26.074720    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" containerID="cri-o://f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63" gracePeriod=30
Sep 18 19:53:26.074979 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:26.074774    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kube-rbac-proxy" containerID="cri-o://3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921" gracePeriod=30
Sep 18 19:53:26.115897 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:26.115864    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:26.740565 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:26.740532    2559 generic.go:356] "Generic (PLEG): container finished" podID="825423dc-e875-4452-837c-aa529baafdb8" containerID="3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921" exitCode=2
Sep 18 19:53:26.740741 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:26.740581    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" event={"ID":"825423dc-e875-4452-837c-aa529baafdb8","Type":"ContainerDied","Data":"3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921"}
Sep 18 19:53:28.403437 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:28.403388    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Sep 18 19:53:28.410232 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:28.410205    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Sep 18 19:53:30.600635 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.600612    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:53:30.758512 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.758487    2559 generic.go:356] "Generic (PLEG): container finished" podID="825423dc-e875-4452-837c-aa529baafdb8" containerID="f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63" exitCode=0
Sep 18 19:53:30.758636 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.758537    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" event={"ID":"825423dc-e875-4452-837c-aa529baafdb8","Type":"ContainerDied","Data":"f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63"}
Sep 18 19:53:30.758636 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.758588    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn" event={"ID":"825423dc-e875-4452-837c-aa529baafdb8","Type":"ContainerDied","Data":"9aa8ee7452263dbf3c1450ce4ecd0877aad7f50d2e5f46fdcf011f8e72957a8b"}
Sep 18 19:53:30.758636 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.758614    2559 scope.go:117] "RemoveContainer" containerID="3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921"
Sep 18 19:53:30.758636 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.758614    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"
Sep 18 19:53:30.766044 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.766020    2559 scope.go:117] "RemoveContainer" containerID="f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63"
Sep 18 19:53:30.772922 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.772907    2559 scope.go:117] "RemoveContainer" containerID="26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd"
Sep 18 19:53:30.779532 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.779518    2559 scope.go:117] "RemoveContainer" containerID="3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921"
Sep 18 19:53:30.779768 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:30.779752    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921\": container with ID starting with 3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921 not found: ID does not exist" containerID="3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921"
Sep 18 19:53:30.779825 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.779775    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921"} err="failed to get container status \"3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921\": rpc error: code = NotFound desc = could not find container \"3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921\": container with ID starting with 3b623ef8051d5f5e4b2f7ce8da8ba4716d199876fb30dd756b7fe95ba2669921 not found: ID does not exist"
Sep 18 19:53:30.779825 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.779790    2559 scope.go:117] "RemoveContainer" containerID="f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63"
Sep 18 19:53:30.780050 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:30.780033    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63\": container with ID starting with f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63 not found: ID does not exist" containerID="f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63"
Sep 18 19:53:30.780086 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.780056    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63"} err="failed to get container status \"f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63\": rpc error: code = NotFound desc = could not find container \"f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63\": container with ID starting with f819988a4d050ca68c0ee7bcb50f3c6b0e12c1c6c1bbdd9b036d989711bb6a63 not found: ID does not exist"
Sep 18 19:53:30.780086 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.780071    2559 scope.go:117] "RemoveContainer" containerID="26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd"
Sep 18 19:53:30.780298 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:30.780284    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd\": container with ID starting with 26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd not found: ID does not exist" containerID="26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd"
Sep 18 19:53:30.780345 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.780301    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd"} err="failed to get container status \"26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd\": rpc error: code = NotFound desc = could not find container \"26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd\": container with ID starting with 26b8a4ab24390f4a40fdeeb0ec0197444b532e36d6e3d4fee82bac706365dffd not found: ID does not exist"
Sep 18 19:53:30.791084 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791067    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-kube-rbac-proxy-sar-config\") pod \"825423dc-e875-4452-837c-aa529baafdb8\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") "
Sep 18 19:53:30.791149 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791101    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-cabundle-cert\") pod \"825423dc-e875-4452-837c-aa529baafdb8\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") "
Sep 18 19:53:30.791149 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791129    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/825423dc-e875-4452-837c-aa529baafdb8-kserve-provision-location\") pod \"825423dc-e875-4452-837c-aa529baafdb8\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") "
Sep 18 19:53:30.791299 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791282    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlztc\" (UniqueName: \"kubernetes.io/projected/825423dc-e875-4452-837c-aa529baafdb8-kube-api-access-wlztc\") pod \"825423dc-e875-4452-837c-aa529baafdb8\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") "
Sep 18 19:53:30.791387 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791366    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/825423dc-e875-4452-837c-aa529baafdb8-proxy-tls\") pod \"825423dc-e875-4452-837c-aa529baafdb8\" (UID: \"825423dc-e875-4452-837c-aa529baafdb8\") "
Sep 18 19:53:30.791491 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791399    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825423dc-e875-4452-837c-aa529baafdb8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "825423dc-e875-4452-837c-aa529baafdb8" (UID: "825423dc-e875-4452-837c-aa529baafdb8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:30.791491 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791427    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "825423dc-e875-4452-837c-aa529baafdb8" (UID: "825423dc-e875-4452-837c-aa529baafdb8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:30.791491 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791440    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "825423dc-e875-4452-837c-aa529baafdb8" (UID: "825423dc-e875-4452-837c-aa529baafdb8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:30.791623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791549    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:30.791623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791560    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/825423dc-e875-4452-837c-aa529baafdb8-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:30.791623 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.791570    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/825423dc-e875-4452-837c-aa529baafdb8-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:30.793071 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.793044    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825423dc-e875-4452-837c-aa529baafdb8-kube-api-access-wlztc" (OuterVolumeSpecName: "kube-api-access-wlztc") pod "825423dc-e875-4452-837c-aa529baafdb8" (UID: "825423dc-e875-4452-837c-aa529baafdb8"). InnerVolumeSpecName "kube-api-access-wlztc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:30.793213 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.793185    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/825423dc-e875-4452-837c-aa529baafdb8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "825423dc-e875-4452-837c-aa529baafdb8" (UID: "825423dc-e875-4452-837c-aa529baafdb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:30.892311 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.892284    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlztc\" (UniqueName: \"kubernetes.io/projected/825423dc-e875-4452-837c-aa529baafdb8-kube-api-access-wlztc\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:30.892311 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:30.892306    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/825423dc-e875-4452-837c-aa529baafdb8-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:31.080167 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.080136    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"]
Sep 18 19:53:31.083721 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.083698    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-585bd9bcc8-9fltn"]
Sep 18 19:53:31.260009 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.259982    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:31.310752 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.310728    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"]
Sep 18 19:53:31.311002 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.310990    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container"
Sep 18 19:53:31.311055 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.311003    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container"
Sep 18 19:53:31.311055 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.311018    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kube-rbac-proxy"
Sep 18 19:53:31.311055 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.311023    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kube-rbac-proxy"
Sep 18 19:53:31.311055 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.311032    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="storage-initializer"
Sep 18 19:53:31.311055 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.311038    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="storage-initializer"
Sep 18 19:53:31.311208 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.311095    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kube-rbac-proxy"
Sep 18 19:53:31.311208 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.311105    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="825423dc-e875-4452-837c-aa529baafdb8" containerName="kserve-container"
Sep 18 19:53:31.314307 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.314293    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.316842 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.316814    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 18 19:53:31.316842 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.316835    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 18 19:53:31.326627 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.326608    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"]
Sep 18 19:53:31.346097 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.346077    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:31.380034 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.380015    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"]
Sep 18 19:53:31.382799 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.382781    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.382979 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.382942    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:53:31.384459 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.384440    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:53:31.384596 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.384579    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 18 19:53:31.393281 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.393261    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"]
Sep 18 19:53:31.469762 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.469739    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="825423dc-e875-4452-837c-aa529baafdb8" path="/var/lib/kubelet/pods/825423dc-e875-4452-837c-aa529baafdb8/volumes"
Sep 18 19:53:31.496129 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496103    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.496231 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496169    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.496231 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496203    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.496339 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496238    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.496339 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496293    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.496402 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496335    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7qm\" (UniqueName: \"kubernetes.io/projected/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-api-access-dv7qm\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.496402 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496361    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mtc\" (UniqueName: \"kubernetes.io/projected/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-api-access-g5mtc\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.496402 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80df351c-06a1-4b45-9a4e-ea78e9296626-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.496529 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496400    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.496529 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.496419    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.596748 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.596703    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.596748 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.596729    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.596748 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.596756    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.596994 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.596774    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.596994 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:31.596862    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 18 19:53:31.596994 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.596908    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7qm\" (UniqueName: \"kubernetes.io/projected/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-api-access-dv7qm\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.596994 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:31.596918    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls podName:80df351c-06a1-4b45-9a4e-ea78e9296626 nodeName:}" failed. No retries permitted until 2026-09-18 19:53:32.096899533 +0000 UTC m=+903.059632265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls") pod "isvc-canary-stop-predictor-5d5b899946-b7bt5" (UID: "80df351c-06a1-4b45-9a4e-ea78e9296626") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 18 19:53:31.596994 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.596948    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mtc\" (UniqueName: \"kubernetes.io/projected/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-api-access-g5mtc\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.597318 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.596997    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80df351c-06a1-4b45-9a4e-ea78e9296626-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.597318 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597033    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.597318 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597062    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.597318 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597089    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.597318 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:31.597160    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-serving-cert: secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 18 19:53:31.597318 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597159    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.597318 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:31.597228    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls podName:4ae1660d-14f3-449d-9be0-f0cfa8fa142a nodeName:}" failed. No retries permitted until 2026-09-18 19:53:32.097211463 +0000 UTC m=+903.059944202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls") pod "isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" (UID: "4ae1660d-14f3-449d-9be0-f0cfa8fa142a") : secret "isvc-canary-stop-v2-predictor-serving-cert" not found
Sep 18 19:53:31.597633 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597444    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80df351c-06a1-4b45-9a4e-ea78e9296626-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.597633 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.597633 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597605    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.597763 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597747    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-cabundle-cert\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.597804 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.597747    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.605472 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.605439    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7qm\" (UniqueName: \"kubernetes.io/projected/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-api-access-dv7qm\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:31.605813 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.605610    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mtc\" (UniqueName: \"kubernetes.io/projected/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-api-access-g5mtc\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:31.756114 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.756084    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:31.854805 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:31.854746    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:32.100896 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.100851    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:32.101036 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.100909    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:32.103596 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.103559    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-5f87c564d7-c88ht\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:32.103687 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.103604    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls\") pod \"isvc-canary-stop-predictor-5d5b899946-b7bt5\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:32.224299 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.224278    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:32.294326 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.294299    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:32.498658 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.498634    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"]
Sep 18 19:53:32.500971 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:53:32.500938    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae1660d_14f3_449d_9be0_f0cfa8fa142a.slice/crio-e804d9a755d44c682ca411b8dfce31d14e16c9630e9ea2093e8a1b2379676233 WatchSource:0}: Error finding container e804d9a755d44c682ca411b8dfce31d14e16c9630e9ea2093e8a1b2379676233: Status 404 returned error can't find the container with id e804d9a755d44c682ca411b8dfce31d14e16c9630e9ea2093e8a1b2379676233
Sep 18 19:53:32.527608 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.523565    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:53:32.569732 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.569702    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"]
Sep 18 19:53:32.572180 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:53:32.572152    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80df351c_06a1_4b45_9a4e_ea78e9296626.slice/crio-2f9a63f19e9365955bdfb22a19866b9b3149608d9531df097a887b6eae76fc99 WatchSource:0}: Error finding container 2f9a63f19e9365955bdfb22a19866b9b3149608d9531df097a887b6eae76fc99: Status 404 returned error can't find the container with id 2f9a63f19e9365955bdfb22a19866b9b3149608d9531df097a887b6eae76fc99
Sep 18 19:53:32.770873 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.770775    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" event={"ID":"4ae1660d-14f3-449d-9be0-f0cfa8fa142a","Type":"ContainerStarted","Data":"eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d"}
Sep 18 19:53:32.770873 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.770856    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" event={"ID":"4ae1660d-14f3-449d-9be0-f0cfa8fa142a","Type":"ContainerStarted","Data":"e804d9a755d44c682ca411b8dfce31d14e16c9630e9ea2093e8a1b2379676233"}
Sep 18 19:53:32.772632 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.772599    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" event={"ID":"80df351c-06a1-4b45-9a4e-ea78e9296626","Type":"ContainerStarted","Data":"b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7"}
Sep 18 19:53:32.772749 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:32.772669    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" event={"ID":"80df351c-06a1-4b45-9a4e-ea78e9296626","Type":"ContainerStarted","Data":"2f9a63f19e9365955bdfb22a19866b9b3149608d9531df097a887b6eae76fc99"}
Sep 18 19:53:36.789569 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:36.789532    2559 generic.go:356] "Generic (PLEG): container finished" podID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerID="b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7" exitCode=0
Sep 18 19:53:36.789946 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:36.789606    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" event={"ID":"80df351c-06a1-4b45-9a4e-ea78e9296626","Type":"ContainerDied","Data":"b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7"}
Sep 18 19:53:36.791281 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:36.791258    2559 generic.go:356] "Generic (PLEG): container finished" podID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerID="eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d" exitCode=0
Sep 18 19:53:36.791380 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:36.791302    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" event={"ID":"4ae1660d-14f3-449d-9be0-f0cfa8fa142a","Type":"ContainerDied","Data":"eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d"}
Sep 18 19:53:37.797146 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:37.797108    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" event={"ID":"4ae1660d-14f3-449d-9be0-f0cfa8fa142a","Type":"ContainerStarted","Data":"82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033"}
Sep 18 19:53:37.797610 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:37.797180    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" event={"ID":"4ae1660d-14f3-449d-9be0-f0cfa8fa142a","Type":"ContainerStarted","Data":"df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326"}
Sep 18 19:53:37.797610 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:37.797401    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:37.799155 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:37.799129    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" event={"ID":"80df351c-06a1-4b45-9a4e-ea78e9296626","Type":"ContainerStarted","Data":"01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6"}
Sep 18 19:53:37.799270 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:37.799177    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" event={"ID":"80df351c-06a1-4b45-9a4e-ea78e9296626","Type":"ContainerStarted","Data":"e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4"}
Sep 18 19:53:37.799368 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:37.799354    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:37.817704 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:37.817662    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podStartSLOduration=3.242585483 podStartE2EDuration="6.817647722s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.575062239s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:37.816485525 +0000 UTC m=+908.779218264" watchObservedRunningTime="2026-09-18 19:53:37.817647722 +0000 UTC m=+908.780380459"
Sep 18 19:53:37.835236 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:37.835194    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podStartSLOduration=3.124144345 podStartE2EDuration="6.835180886s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711036541s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:53:37.833763257 +0000 UTC m=+908.796495994" watchObservedRunningTime="2026-09-18 19:53:37.835180886 +0000 UTC m=+908.797913624"
Sep 18 19:53:38.803050 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:38.803019    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:38.803432 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:38.803088    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:38.803860 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:38.803836    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:53:38.803912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:38.803842    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:53:39.807056 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:39.807005    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:53:39.807459 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:39.807085    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:53:41.155656 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:41.155621    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:41.421908 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:41.421829    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:44.812551 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:44.812521    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:53:44.813093 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:44.812607    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:53:44.813093 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:44.813023    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:53:44.813093 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:44.813061    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:53:51.237261 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.237232    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:51.319429 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.319394    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"]
Sep 18 19:53:51.319722 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.319695    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" containerID="cri-o://a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418" gracePeriod=30
Sep 18 19:53:51.319782 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.319727    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kube-rbac-proxy" containerID="cri-o://12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51" gracePeriod=30
Sep 18 19:53:51.348347 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.348315    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:51.354010 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.353985    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"]
Sep 18 19:53:51.354418 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.354253    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" containerID="cri-o://c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f" gracePeriod=30
Sep 18 19:53:51.354418 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.354288    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kube-rbac-proxy" containerID="cri-o://0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8" gracePeriod=30
Sep 18 19:53:51.446438 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.446402    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:51.548542 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.548434    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:51.612226 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.612196    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:51.856003 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.855915    2559 generic.go:356] "Generic (PLEG): container finished" podID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerID="12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51" exitCode=2
Sep 18 19:53:51.856164 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.855999    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" event={"ID":"292307d8-3fb5-4b4a-9531-fa2f622588c4","Type":"ContainerDied","Data":"12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51"}
Sep 18 19:53:51.858023 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.857997    2559 generic.go:356] "Generic (PLEG): container finished" podID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerID="0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8" exitCode=2
Sep 18 19:53:51.858140 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.858027    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" event={"ID":"a564cac9-6b13-4726-8f8f-ebd8f7af0b57","Type":"ContainerDied","Data":"0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8"}
Sep 18 19:53:51.984691 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:51.984658    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:53:52.518211 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:52.518177    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Sep 18 19:53:52.523533 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:52.523508    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Sep 18 19:53:53.431978 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:53.431934    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Sep 18 19:53:53.437656 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:53.437631    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Sep 18 19:53:54.813308 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:54.813273    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:53:54.813672 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:54.813280    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:53:55.401630 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.401605    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:53:55.475348 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.475322    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-cabundle-cert\") pod \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") "
Sep 18 19:53:55.475498 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.475363    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwdkd\" (UniqueName: \"kubernetes.io/projected/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-api-access-bwdkd\") pod \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") "
Sep 18 19:53:55.475498 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.475402    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kserve-provision-location\") pod \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") "
Sep 18 19:53:55.475498 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.475423    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls\") pod \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") "
Sep 18 19:53:55.475498 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.475478    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-rbac-proxy-sar-config\") pod \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\" (UID: \"a564cac9-6b13-4726-8f8f-ebd8f7af0b57\") "
Sep 18 19:53:55.475773 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.475746    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a564cac9-6b13-4726-8f8f-ebd8f7af0b57" (UID: "a564cac9-6b13-4726-8f8f-ebd8f7af0b57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:55.475829 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.475764    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a564cac9-6b13-4726-8f8f-ebd8f7af0b57" (UID: "a564cac9-6b13-4726-8f8f-ebd8f7af0b57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:55.475893 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.475869    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a564cac9-6b13-4726-8f8f-ebd8f7af0b57" (UID: "a564cac9-6b13-4726-8f8f-ebd8f7af0b57"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:55.477531 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.477510    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-api-access-bwdkd" (OuterVolumeSpecName: "kube-api-access-bwdkd") pod "a564cac9-6b13-4726-8f8f-ebd8f7af0b57" (UID: "a564cac9-6b13-4726-8f8f-ebd8f7af0b57"). InnerVolumeSpecName "kube-api-access-bwdkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:55.477625 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.477607    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a564cac9-6b13-4726-8f8f-ebd8f7af0b57" (UID: "a564cac9-6b13-4726-8f8f-ebd8f7af0b57"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:55.576828 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.576753    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:55.576828 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.576779    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:55.576828 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.576794    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwdkd\" (UniqueName: \"kubernetes.io/projected/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kube-api-access-bwdkd\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:55.576828 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.576806    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:55.576828 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.576817    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a564cac9-6b13-4726-8f8f-ebd8f7af0b57-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:55.874532 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.874456    2559 generic.go:356] "Generic (PLEG): container finished" podID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerID="c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f" exitCode=0
Sep 18 19:53:55.874532 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.874502    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" event={"ID":"a564cac9-6b13-4726-8f8f-ebd8f7af0b57","Type":"ContainerDied","Data":"c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f"}
Sep 18 19:53:55.874532 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.874513    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"
Sep 18 19:53:55.875008 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.874552    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx" event={"ID":"a564cac9-6b13-4726-8f8f-ebd8f7af0b57","Type":"ContainerDied","Data":"3c84ecbbd17ccf7dfc4be97a082f0c2703cd8332aa6ea73bee852f3f3b50c9e1"}
Sep 18 19:53:55.875008 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.874578    2559 scope.go:117] "RemoveContainer" containerID="0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8"
Sep 18 19:53:55.885133 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.885113    2559 scope.go:117] "RemoveContainer" containerID="c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f"
Sep 18 19:53:55.892518 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.892502    2559 scope.go:117] "RemoveContainer" containerID="a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15"
Sep 18 19:53:55.898629 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.898604    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"]
Sep 18 19:53:55.900131 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.899998    2559 scope.go:117] "RemoveContainer" containerID="0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8"
Sep 18 19:53:55.900318 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:55.900293    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8\": container with ID starting with 0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8 not found: ID does not exist" containerID="0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8"
Sep 18 19:53:55.900467 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.900327    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8"} err="failed to get container status \"0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8\": rpc error: code = NotFound desc = could not find container \"0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8\": container with ID starting with 0531ba2f42319c729ea211df9b9abe286b98511bb60c0c7f1684799ed167a9e8 not found: ID does not exist"
Sep 18 19:53:55.900467 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.900350    2559 scope.go:117] "RemoveContainer" containerID="c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f"
Sep 18 19:53:55.900698 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:55.900671    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f\": container with ID starting with c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f not found: ID does not exist" containerID="c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f"
Sep 18 19:53:55.900753 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.900699    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f"} err="failed to get container status \"c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f\": rpc error: code = NotFound desc = could not find container \"c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f\": container with ID starting with c13269d437549a1db1bc5173c20a1fadbf72e981dc80aed7538fbb6b65e0a79f not found: ID does not exist"
Sep 18 19:53:55.900753 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.900716    2559 scope.go:117] "RemoveContainer" containerID="a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15"
Sep 18 19:53:55.900999 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:55.900978    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15\": container with ID starting with a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15 not found: ID does not exist" containerID="a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15"
Sep 18 19:53:55.901045 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.901006    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15"} err="failed to get container status \"a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15\": rpc error: code = NotFound desc = could not find container \"a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15\": container with ID starting with a30cf7d532cdf0e64a7c9ff9056315888f2d06999c067e148672f307c97d5b15 not found: ID does not exist"
Sep 18 19:53:55.901736 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:55.901719    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-2f453-predictor-6f4b8856df-5qhrx"]
Sep 18 19:53:56.669584 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.669562    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:53:56.785818 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.785745    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-cabundle-cert\") pod \"292307d8-3fb5-4b4a-9531-fa2f622588c4\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") "
Sep 18 19:53:56.785818 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.785792    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-rbac-proxy-sar-config\") pod \"292307d8-3fb5-4b4a-9531-fa2f622588c4\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") "
Sep 18 19:53:56.785818 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.785817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292307d8-3fb5-4b4a-9531-fa2f622588c4-kserve-provision-location\") pod \"292307d8-3fb5-4b4a-9531-fa2f622588c4\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") "
Sep 18 19:53:56.786138 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.785853    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkbsm\" (UniqueName: \"kubernetes.io/projected/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-api-access-jkbsm\") pod \"292307d8-3fb5-4b4a-9531-fa2f622588c4\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") "
Sep 18 19:53:56.786138 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.785873    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292307d8-3fb5-4b4a-9531-fa2f622588c4-proxy-tls\") pod \"292307d8-3fb5-4b4a-9531-fa2f622588c4\" (UID: \"292307d8-3fb5-4b4a-9531-fa2f622588c4\") "
Sep 18 19:53:56.786138 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.786104    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "292307d8-3fb5-4b4a-9531-fa2f622588c4" (UID: "292307d8-3fb5-4b4a-9531-fa2f622588c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:56.786260 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.786154    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292307d8-3fb5-4b4a-9531-fa2f622588c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "292307d8-3fb5-4b4a-9531-fa2f622588c4" (UID: "292307d8-3fb5-4b4a-9531-fa2f622588c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:53:56.786260 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.786218    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "292307d8-3fb5-4b4a-9531-fa2f622588c4" (UID: "292307d8-3fb5-4b4a-9531-fa2f622588c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:53:56.788013 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.787995    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-api-access-jkbsm" (OuterVolumeSpecName: "kube-api-access-jkbsm") pod "292307d8-3fb5-4b4a-9531-fa2f622588c4" (UID: "292307d8-3fb5-4b4a-9531-fa2f622588c4"). InnerVolumeSpecName "kube-api-access-jkbsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:53:56.788113 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.788093    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292307d8-3fb5-4b4a-9531-fa2f622588c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "292307d8-3fb5-4b4a-9531-fa2f622588c4" (UID: "292307d8-3fb5-4b4a-9531-fa2f622588c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:53:56.880297 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.880265    2559 generic.go:356] "Generic (PLEG): container finished" podID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerID="a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418" exitCode=0
Sep 18 19:53:56.880683 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.880325    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"
Sep 18 19:53:56.880683 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.880355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" event={"ID":"292307d8-3fb5-4b4a-9531-fa2f622588c4","Type":"ContainerDied","Data":"a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418"}
Sep 18 19:53:56.880683 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.880420    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9" event={"ID":"292307d8-3fb5-4b4a-9531-fa2f622588c4","Type":"ContainerDied","Data":"06df14b77ee766ab5a6b03a0c8762a89159263493de9c23a174df43eb9274d24"}
Sep 18 19:53:56.880683 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.880440    2559 scope.go:117] "RemoveContainer" containerID="12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51"
Sep 18 19:53:56.889358 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.888280    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:56.889358 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.888312    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:56.889358 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.888326    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292307d8-3fb5-4b4a-9531-fa2f622588c4-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:56.889358 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.888361    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkbsm\" (UniqueName: \"kubernetes.io/projected/292307d8-3fb5-4b4a-9531-fa2f622588c4-kube-api-access-jkbsm\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:56.889358 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.888375    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/292307d8-3fb5-4b4a-9531-fa2f622588c4-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:53:56.889837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.889533    2559 scope.go:117] "RemoveContainer" containerID="a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418"
Sep 18 19:53:56.898011 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.897992    2559 scope.go:117] "RemoveContainer" containerID="90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420"
Sep 18 19:53:56.905313 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.905289    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"]
Sep 18 19:53:56.905714 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.905702    2559 scope.go:117] "RemoveContainer" containerID="12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51"
Sep 18 19:53:56.905978 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:56.905948    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51\": container with ID starting with 12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51 not found: ID does not exist" containerID="12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51"
Sep 18 19:53:56.906023 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.905986    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51"} err="failed to get container status \"12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51\": rpc error: code = NotFound desc = could not find container \"12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51\": container with ID starting with 12fed75fbe4e9b64f8c531be8a6f9c65247602226054ca89daa2cd1c622dec51 not found: ID does not exist"
Sep 18 19:53:56.906023 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.906003    2559 scope.go:117] "RemoveContainer" containerID="a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418"
Sep 18 19:53:56.906249 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:56.906234    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418\": container with ID starting with a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418 not found: ID does not exist" containerID="a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418"
Sep 18 19:53:56.906295 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.906252    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418"} err="failed to get container status \"a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418\": rpc error: code = NotFound desc = could not find container \"a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418\": container with ID starting with a074487a5d347df9c1306e1c0d7efbaaa2e95686424e7732afcd912a2daab418 not found: ID does not exist"
Sep 18 19:53:56.906295 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.906266    2559 scope.go:117] "RemoveContainer" containerID="90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420"
Sep 18 19:53:56.906467 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:53:56.906452    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420\": container with ID starting with 90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420 not found: ID does not exist" containerID="90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420"
Sep 18 19:53:56.906502 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.906470    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420"} err="failed to get container status \"90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420\": rpc error: code = NotFound desc = could not find container \"90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420\": container with ID starting with 90a7b6807060e39c1c83000cd875707377c5224793879bd8d50337603b039420 not found: ID does not exist"
Sep 18 19:53:56.909200 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:56.909181    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-2f453-predictor-cbd89768c-krst9"]
Sep 18 19:53:57.469828 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:57.469796    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" path="/var/lib/kubelet/pods/292307d8-3fb5-4b4a-9531-fa2f622588c4/volumes"
Sep 18 19:53:57.470320 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:53:57.470305    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" path="/var/lib/kubelet/pods/a564cac9-6b13-4726-8f8f-ebd8f7af0b57/volumes"
Sep 18 19:54:04.813954 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:04.813909    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:54:04.813954 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:04.813920    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:54:14.813584 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:14.813548    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:54:14.813990 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:14.813554    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:54:24.813182 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:24.813145    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:54:24.813182 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:24.813167    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:54:34.813006 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:34.812953    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:54:34.813370 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:34.812978    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:54:44.813849 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:44.813759    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Sep 18 19:54:44.814318 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:44.813757    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:54:54.814232 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:54.814199    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:54:54.814716 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:54.814444    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:54:54.937630 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:54.937600    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"]
Sep 18 19:54:55.102213 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:55.102103    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container" containerID="cri-o://e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4" gracePeriod=30
Sep 18 19:54:55.102213 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:55.102152    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kube-rbac-proxy" containerID="cri-o://01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6" gracePeriod=30
Sep 18 19:54:56.108038 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:56.108007    2559 generic.go:356] "Generic (PLEG): container finished" podID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerID="01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6" exitCode=2
Sep 18 19:54:56.108456 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:56.108060    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" event={"ID":"80df351c-06a1-4b45-9a4e-ea78e9296626","Type":"ContainerDied","Data":"01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6"}
Sep 18 19:54:57.168055 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:57.168021    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:57.265890 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:57.265867    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:58.945184 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:58.945159    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:54:59.033262 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033207    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-rbac-proxy-sar-config\") pod \"80df351c-06a1-4b45-9a4e-ea78e9296626\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") "
Sep 18 19:54:59.033262 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033242    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-cabundle-cert\") pod \"80df351c-06a1-4b45-9a4e-ea78e9296626\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") "
Sep 18 19:54:59.033262 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033258    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv7qm\" (UniqueName: \"kubernetes.io/projected/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-api-access-dv7qm\") pod \"80df351c-06a1-4b45-9a4e-ea78e9296626\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") "
Sep 18 19:54:59.033473 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033407    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls\") pod \"80df351c-06a1-4b45-9a4e-ea78e9296626\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") "
Sep 18 19:54:59.033473 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033454    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80df351c-06a1-4b45-9a4e-ea78e9296626-kserve-provision-location\") pod \"80df351c-06a1-4b45-9a4e-ea78e9296626\" (UID: \"80df351c-06a1-4b45-9a4e-ea78e9296626\") "
Sep 18 19:54:59.033575 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033514    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "80df351c-06a1-4b45-9a4e-ea78e9296626" (UID: "80df351c-06a1-4b45-9a4e-ea78e9296626"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:59.033575 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033535    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "80df351c-06a1-4b45-9a4e-ea78e9296626" (UID: "80df351c-06a1-4b45-9a4e-ea78e9296626"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:54:59.033715 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033699    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:54:59.033767 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033721    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/80df351c-06a1-4b45-9a4e-ea78e9296626-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:54:59.033804 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.033772    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80df351c-06a1-4b45-9a4e-ea78e9296626-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "80df351c-06a1-4b45-9a4e-ea78e9296626" (UID: "80df351c-06a1-4b45-9a4e-ea78e9296626"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:54:59.035320 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.035298    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-api-access-dv7qm" (OuterVolumeSpecName: "kube-api-access-dv7qm") pod "80df351c-06a1-4b45-9a4e-ea78e9296626" (UID: "80df351c-06a1-4b45-9a4e-ea78e9296626"). InnerVolumeSpecName "kube-api-access-dv7qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:54:59.035417 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.035334    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80df351c-06a1-4b45-9a4e-ea78e9296626" (UID: "80df351c-06a1-4b45-9a4e-ea78e9296626"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:54:59.120947 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.120921    2559 generic.go:356] "Generic (PLEG): container finished" podID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerID="e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4" exitCode=0
Sep 18 19:54:59.121074 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.120957    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" event={"ID":"80df351c-06a1-4b45-9a4e-ea78e9296626","Type":"ContainerDied","Data":"e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4"}
Sep 18 19:54:59.121074 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.120993    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"
Sep 18 19:54:59.121074 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.121042    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5" event={"ID":"80df351c-06a1-4b45-9a4e-ea78e9296626","Type":"ContainerDied","Data":"2f9a63f19e9365955bdfb22a19866b9b3149608d9531df097a887b6eae76fc99"}
Sep 18 19:54:59.121225 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.121082    2559 scope.go:117] "RemoveContainer" containerID="01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6"
Sep 18 19:54:59.133856 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.133840    2559 scope.go:117] "RemoveContainer" containerID="e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4"
Sep 18 19:54:59.134070 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.134054    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv7qm\" (UniqueName: \"kubernetes.io/projected/80df351c-06a1-4b45-9a4e-ea78e9296626-kube-api-access-dv7qm\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:54:59.134157 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.134075    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80df351c-06a1-4b45-9a4e-ea78e9296626-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:54:59.134157 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.134097    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/80df351c-06a1-4b45-9a4e-ea78e9296626-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:54:59.140584 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.140566    2559 scope.go:117] "RemoveContainer" containerID="b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7"
Sep 18 19:54:59.143492 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.143473    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"]
Sep 18 19:54:59.148817 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.148349    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5d5b899946-b7bt5"]
Sep 18 19:54:59.149442 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.149400    2559 scope.go:117] "RemoveContainer" containerID="01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6"
Sep 18 19:54:59.149849 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:54:59.149832    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6\": container with ID starting with 01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6 not found: ID does not exist" containerID="01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6"
Sep 18 19:54:59.149912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.149865    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6"} err="failed to get container status \"01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6\": rpc error: code = NotFound desc = could not find container \"01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6\": container with ID starting with 01f204d060f295072e9169929fb3569565fb22cad1a5e85b7cdebe25db3b61a6 not found: ID does not exist"
Sep 18 19:54:59.149912 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.149881    2559 scope.go:117] "RemoveContainer" containerID="e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4"
Sep 18 19:54:59.150188 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:54:59.150169    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4\": container with ID starting with e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4 not found: ID does not exist" containerID="e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4"
Sep 18 19:54:59.150241 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.150198    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4"} err="failed to get container status \"e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4\": rpc error: code = NotFound desc = could not find container \"e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4\": container with ID starting with e758d0f381ac0f2fdce449d84f95bbb96f97955cd54ea37a396efda1eda9cba4 not found: ID does not exist"
Sep 18 19:54:59.150241 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.150211    2559 scope.go:117] "RemoveContainer" containerID="b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7"
Sep 18 19:54:59.150463 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:54:59.150447    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7\": container with ID starting with b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7 not found: ID does not exist" containerID="b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7"
Sep 18 19:54:59.150504 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.150468    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7"} err="failed to get container status \"b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7\": rpc error: code = NotFound desc = could not find container \"b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7\": container with ID starting with b2f45deafbee84047a599ef0e4e73bea9b751fe84d553019c319b42877c4ecb7 not found: ID does not exist"
Sep 18 19:54:59.237160 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.237119    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:54:59.470369 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:54:59.470344    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" path="/var/lib/kubelet/pods/80df351c-06a1-4b45-9a4e-ea78e9296626/volumes"
Sep 18 19:55:01.388825 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:01.388792    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"]
Sep 18 19:55:01.389275 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:01.389053    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" containerID="cri-o://df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326" gracePeriod=30
Sep 18 19:55:01.389275 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:01.389117    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kube-rbac-proxy" containerID="cri-o://82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033" gracePeriod=30
Sep 18 19:55:01.394065 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:01.394041    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:01.503264 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:01.503238    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:02.134821 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:02.134791    2559 generic.go:356] "Generic (PLEG): container finished" podID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerID="82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033" exitCode=2
Sep 18 19:55:02.134982 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:02.134863    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" event={"ID":"4ae1660d-14f3-449d-9be0-f0cfa8fa142a","Type":"ContainerDied","Data":"82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033"}
Sep 18 19:55:04.808016 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:04.807955    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Sep 18 19:55:04.813189 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:04.813161    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Sep 18 19:55:05.123545 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.123523    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:55:05.148699 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.148671    2559 generic.go:356] "Generic (PLEG): container finished" podID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerID="df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326" exitCode=0
Sep 18 19:55:05.148821 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.148717    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"
Sep 18 19:55:05.148821 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.148717    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" event={"ID":"4ae1660d-14f3-449d-9be0-f0cfa8fa142a","Type":"ContainerDied","Data":"df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326"}
Sep 18 19:55:05.148938 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.148824    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht" event={"ID":"4ae1660d-14f3-449d-9be0-f0cfa8fa142a","Type":"ContainerDied","Data":"e804d9a755d44c682ca411b8dfce31d14e16c9630e9ea2093e8a1b2379676233"}
Sep 18 19:55:05.148938 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.148867    2559 scope.go:117] "RemoveContainer" containerID="82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033"
Sep 18 19:55:05.156347 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.156330    2559 scope.go:117] "RemoveContainer" containerID="df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326"
Sep 18 19:55:05.163121 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.163100    2559 scope.go:117] "RemoveContainer" containerID="eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d"
Sep 18 19:55:05.169537 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.169523    2559 scope.go:117] "RemoveContainer" containerID="82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033"
Sep 18 19:55:05.169776 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:55:05.169756    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033\": container with ID starting with 82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033 not found: ID does not exist" containerID="82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033"
Sep 18 19:55:05.169858 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.169783    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033"} err="failed to get container status \"82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033\": rpc error: code = NotFound desc = could not find container \"82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033\": container with ID starting with 82e7fba01d75b2c032a218b36e7a499e16b47af2b4fe4cd12e1162c9a402e033 not found: ID does not exist"
Sep 18 19:55:05.169858 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.169799    2559 scope.go:117] "RemoveContainer" containerID="df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326"
Sep 18 19:55:05.170042 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:55:05.170024    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326\": container with ID starting with df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326 not found: ID does not exist" containerID="df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326"
Sep 18 19:55:05.170103 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.170047    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326"} err="failed to get container status \"df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326\": rpc error: code = NotFound desc = could not find container \"df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326\": container with ID starting with df2ea8dfe9e6baa86abab5dc23b12631131df9774c01ef3f9973a88489e85326 not found: ID does not exist"
Sep 18 19:55:05.170103 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.170064    2559 scope.go:117] "RemoveContainer" containerID="eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d"
Sep 18 19:55:05.170286 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:55:05.170269    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d\": container with ID starting with eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d not found: ID does not exist" containerID="eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d"
Sep 18 19:55:05.170337 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.170294    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d"} err="failed to get container status \"eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d\": rpc error: code = NotFound desc = could not find container \"eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d\": container with ID starting with eb4cf7acb7951c467f98840499fc6a7c4f17f970a4aa1cef04b18c203c24c32d not found: ID does not exist"
Sep 18 19:55:05.278253 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.278233    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls\") pod \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") "
Sep 18 19:55:05.278327 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.278271    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kserve-provision-location\") pod \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") "
Sep 18 19:55:05.278327 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.278321    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5mtc\" (UniqueName: \"kubernetes.io/projected/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-api-access-g5mtc\") pod \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") "
Sep 18 19:55:05.278415 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.278340    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-rbac-proxy-sar-config\") pod \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") "
Sep 18 19:55:05.278415 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.278373    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-cabundle-cert\") pod \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\" (UID: \"4ae1660d-14f3-449d-9be0-f0cfa8fa142a\") "
Sep 18 19:55:05.278701 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.278677    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ae1660d-14f3-449d-9be0-f0cfa8fa142a" (UID: "4ae1660d-14f3-449d-9be0-f0cfa8fa142a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:55:05.278771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.278714    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ae1660d-14f3-449d-9be0-f0cfa8fa142a" (UID: "4ae1660d-14f3-449d-9be0-f0cfa8fa142a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:05.278771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.278753    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ae1660d-14f3-449d-9be0-f0cfa8fa142a" (UID: "4ae1660d-14f3-449d-9be0-f0cfa8fa142a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:55:05.280287 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.280268    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ae1660d-14f3-449d-9be0-f0cfa8fa142a" (UID: "4ae1660d-14f3-449d-9be0-f0cfa8fa142a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:55:05.280414 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.280396    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-api-access-g5mtc" (OuterVolumeSpecName: "kube-api-access-g5mtc") pod "4ae1660d-14f3-449d-9be0-f0cfa8fa142a" (UID: "4ae1660d-14f3-449d-9be0-f0cfa8fa142a"). InnerVolumeSpecName "kube-api-access-g5mtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:55:05.379420 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.379369    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5mtc\" (UniqueName: \"kubernetes.io/projected/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-api-access-g5mtc\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:55:05.379420 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.379389    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:55:05.379420 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.379398    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:55:05.379420 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.379407    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:55:05.379420 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.379414    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae1660d-14f3-449d-9be0-f0cfa8fa142a-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:55:05.470201 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.470178    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"]
Sep 18 19:55:05.476141 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:05.476118    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-5f87c564d7-c88ht"]
Sep 18 19:55:07.469287 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:07.469253    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" path="/var/lib/kubelet/pods/4ae1660d-14f3-449d-9be0-f0cfa8fa142a/volumes"
Sep 18 19:55:11.564149 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.564119    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:11.595844 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.595819    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"]
Sep 18 19:55:11.596070 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596058    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="storage-initializer"
Sep 18 19:55:11.596119 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596072    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="storage-initializer"
Sep 18 19:55:11.596119 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596082    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container"
Sep 18 19:55:11.596119 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596087    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container"
Sep 18 19:55:11.596119 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596096    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596119 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596102    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596119 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596108    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="storage-initializer"
Sep 18 19:55:11.596119 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596114    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="storage-initializer"
Sep 18 19:55:11.596119 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596119    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596124    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596133    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="storage-initializer"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596138    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="storage-initializer"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596145    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596150    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596160    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596165    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596170    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596175    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596180    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596185    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596193    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="storage-initializer"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596197    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="storage-initializer"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596204    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596209    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596248    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596255    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596260    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kserve-container"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596266    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a564cac9-6b13-4726-8f8f-ebd8f7af0b57" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596271    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="292307d8-3fb5-4b4a-9531-fa2f622588c4" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596275    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596281    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="80df351c-06a1-4b45-9a4e-ea78e9296626" containerName="kube-rbac-proxy"
Sep 18 19:55:11.596350 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.596286    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae1660d-14f3-449d-9be0-f0cfa8fa142a" containerName="kserve-container"
Sep 18 19:55:11.600572 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.600558    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.602888 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.602860    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-81daee-predictor-serving-cert\""
Sep 18 19:55:11.602888 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.602879    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 19:55:11.603122 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.602872    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:55:11.603122 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.603040    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:55:11.603122 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.603042    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\""
Sep 18 19:55:11.603333 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.603147    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-81daee-kube-rbac-proxy-sar-config\""
Sep 18 19:55:11.603521 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.603506    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 19:55:11.608717 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.608694    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"]
Sep 18 19:55:11.622762 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.622743    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.622880 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.622792    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd726\" (UniqueName: \"kubernetes.io/projected/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-api-access-rd726\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.622880 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.622820    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-rbac-proxy-sar-config\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.622880 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.622845    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da54697d-7fcb-44ad-a513-7e4a13b26c58-kserve-provision-location\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.623040 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.622881    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-cabundle-cert\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.723876 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.723849    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.723997 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.723889    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd726\" (UniqueName: \"kubernetes.io/projected/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-api-access-rd726\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.723997 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.723911    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-rbac-proxy-sar-config\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.723997 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.723936    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da54697d-7fcb-44ad-a513-7e4a13b26c58-kserve-provision-location\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.724173 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.724008    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-cabundle-cert\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.724173 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:55:11.724027    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-81daee-predictor-serving-cert: secret "isvc-primary-81daee-predictor-serving-cert" not found
Sep 18 19:55:11.724173 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:55:11.724081    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls podName:da54697d-7fcb-44ad-a513-7e4a13b26c58 nodeName:}" failed. No retries permitted until 2026-09-18 19:55:12.224062831 +0000 UTC m=+1003.186795565 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls") pod "isvc-primary-81daee-predictor-865f44d78-xlh89" (UID: "da54697d-7fcb-44ad-a513-7e4a13b26c58") : secret "isvc-primary-81daee-predictor-serving-cert" not found
Sep 18 19:55:11.724342 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.724326    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da54697d-7fcb-44ad-a513-7e4a13b26c58-kserve-provision-location\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.724697 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.724678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-cabundle-cert\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.724729 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.724678    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-rbac-proxy-sar-config\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:11.733021 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:11.733002    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd726\" (UniqueName: \"kubernetes.io/projected/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-api-access-rd726\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:12.054269 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:12.054242    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:12.226906 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:12.226879    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:12.229209 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:12.229165    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls\") pod \"isvc-primary-81daee-predictor-865f44d78-xlh89\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") " pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:12.511277 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:12.511251    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:12.787454 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:12.787429    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"]
Sep 18 19:55:12.789717 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:55:12.789690    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda54697d_7fcb_44ad_a513_7e4a13b26c58.slice/crio-817406f8652e0c82323f776d71650979e838f4462479fc8b05c5f54708df522b WatchSource:0}: Error finding container 817406f8652e0c82323f776d71650979e838f4462479fc8b05c5f54708df522b: Status 404 returned error can't find the container with id 817406f8652e0c82323f776d71650979e838f4462479fc8b05c5f54708df522b
Sep 18 19:55:13.182108 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:13.182028    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" event={"ID":"da54697d-7fcb-44ad-a513-7e4a13b26c58","Type":"ContainerStarted","Data":"0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864"}
Sep 18 19:55:13.182108 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:13.182093    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" event={"ID":"da54697d-7fcb-44ad-a513-7e4a13b26c58","Type":"ContainerStarted","Data":"817406f8652e0c82323f776d71650979e838f4462479fc8b05c5f54708df522b"}
Sep 18 19:55:17.198574 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:17.198541    2559 generic.go:356] "Generic (PLEG): container finished" podID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerID="0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864" exitCode=0
Sep 18 19:55:17.198910 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:17.198616    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" event={"ID":"da54697d-7fcb-44ad-a513-7e4a13b26c58","Type":"ContainerDied","Data":"0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864"}
Sep 18 19:55:18.205286 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:18.205246    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" event={"ID":"da54697d-7fcb-44ad-a513-7e4a13b26c58","Type":"ContainerStarted","Data":"fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d"}
Sep 18 19:55:18.205691 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:18.205320    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" event={"ID":"da54697d-7fcb-44ad-a513-7e4a13b26c58","Type":"ContainerStarted","Data":"0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07"}
Sep 18 19:55:18.205691 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:18.205517    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:18.224830 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:18.224747    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podStartSLOduration=3.450231492 podStartE2EDuration="7.224731392s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.7744999s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:18.223921817 +0000 UTC m=+1009.186654556" watchObservedRunningTime="2026-09-18 19:55:18.224731392 +0000 UTC m=+1009.187464131"
Sep 18 19:55:19.209810 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:19.209775    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:19.211105 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:19.211069    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:55:20.213797 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:20.213762    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:55:25.218896 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:25.218866    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:55:25.219381 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:25.219354    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:55:31.468733 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:31.468701    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:31.595702 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:31.595672    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:35.220160 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:35.220125    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:55:41.545209 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.545170    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:41.662392 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.662364    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:41.727725 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.727689    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"]
Sep 18 19:55:41.732397 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.732378    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:41.734542 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.734519    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-91b5b-predictor-serving-cert\""
Sep 18 19:55:41.734719 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.734523    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-91b5b-kube-rbac-proxy-sar-config\""
Sep 18 19:55:41.735554 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.735531    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:41.740985 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.740942    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"]
Sep 18 19:55:41.805826 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.805757    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:41.823894 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.823856    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rw2\" (UniqueName: \"kubernetes.io/projected/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-api-access-59rw2\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:41.824037 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.823925    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:41.824037 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.823952    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:41.925032 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.924986    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rw2\" (UniqueName: \"kubernetes.io/projected/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-api-access-59rw2\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:41.925207 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.925145    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:41.925207 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.925184    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:41.925286 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:55:41.925272    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-serving-cert: secret "message-dumper-raw-91b5b-predictor-serving-cert" not found
Sep 18 19:55:41.925352 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:55:41.925341    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls podName:192fe0a2-3850-4a97-bbf4-1c374a74db3c nodeName:}" failed. No retries permitted until 2026-09-18 19:55:42.425318533 +0000 UTC m=+1033.388051274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls") pod "message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" (UID: "192fe0a2-3850-4a97-bbf4-1c374a74db3c") : secret "message-dumper-raw-91b5b-predictor-serving-cert" not found
Sep 18 19:55:41.925810 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.925794    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:41.933493 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:41.933464    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rw2\" (UniqueName: \"kubernetes.io/projected/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-api-access-59rw2\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:42.313539 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:42.313508    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:42.428790 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:42.428744    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:42.431169 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:42.431144    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls\") pod \"message-dumper-raw-91b5b-predictor-598d8989f5-8jv89\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") " pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:42.643622 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:42.643523    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:42.912741 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:42.912640    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"]
Sep 18 19:55:42.915497 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:55:42.915471    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192fe0a2_3850_4a97_bbf4_1c374a74db3c.slice/crio-1fc7f9d5e7968e45738fdef0812952bfe15ed28b51d2575d0f54aba7ae3d8fc1 WatchSource:0}: Error finding container 1fc7f9d5e7968e45738fdef0812952bfe15ed28b51d2575d0f54aba7ae3d8fc1: Status 404 returned error can't find the container with id 1fc7f9d5e7968e45738fdef0812952bfe15ed28b51d2575d0f54aba7ae3d8fc1
Sep 18 19:55:43.296306 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:43.296273    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" event={"ID":"192fe0a2-3850-4a97-bbf4-1c374a74db3c","Type":"ContainerStarted","Data":"1fc7f9d5e7968e45738fdef0812952bfe15ed28b51d2575d0f54aba7ae3d8fc1"}
Sep 18 19:55:44.302124 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:44.302019    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" event={"ID":"192fe0a2-3850-4a97-bbf4-1c374a74db3c","Type":"ContainerStarted","Data":"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3"}
Sep 18 19:55:44.302124 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:44.302090    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" event={"ID":"192fe0a2-3850-4a97-bbf4-1c374a74db3c","Type":"ContainerStarted","Data":"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4"}
Sep 18 19:55:44.302553 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:44.302258    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:44.302553 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:44.302318    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:44.303853 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:44.303833    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:44.318693 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:44.318642    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" podStartSLOduration=2.231000489 podStartE2EDuration="3.318626768s" totalImagesPullingTime="1.087626279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:55:44.317535484 +0000 UTC m=+1035.280268228" watchObservedRunningTime="2026-09-18 19:55:44.318626768 +0000 UTC m=+1035.281359507"
Sep 18 19:55:45.220368 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:45.220332    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:55:51.315170 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:51.315142    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:55:51.736505 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:51.736471    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:51.822938 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:51.822902    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:55:55.219823 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:55:55.219785    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:56:05.219310 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:05.219273    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:56:11.206656 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:11.206616    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:15.219823 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:15.219738    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:56:25.219798 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:25.219751    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:56:35.220397 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:35.220364    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:56:56.786219 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.786165    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:56.816626 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.816595    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"]
Sep 18 19:56:56.820039 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.820020    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.822385 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.822364    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-81daee-predictor-serving-cert\""
Sep 18 19:56:56.823305 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.823284    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-81daee-dockercfg-qlvhs\""
Sep 18 19:56:56.823778 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.823763    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-81daee-kube-rbac-proxy-sar-config\""
Sep 18 19:56:56.823837 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.823777    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-81daee\""
Sep 18 19:56:56.838368 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.838348    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"]
Sep 18 19:56:56.884254 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.884225    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kserve-provision-location\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.884389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.884262    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.884389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.884291    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6lh\" (UniqueName: \"kubernetes.io/projected/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-api-access-xk6lh\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.884389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.884350    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.884389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.884377    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-cabundle-cert\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.985571 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.985540    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.985571 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.985573    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-cabundle-cert\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.985806 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.985629    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kserve-provision-location\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.985806 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.985650    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.985806 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.985671    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6lh\" (UniqueName: \"kubernetes.io/projected/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-api-access-xk6lh\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.985806 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:56:56.985750    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-81daee-predictor-serving-cert: secret "isvc-secondary-81daee-predictor-serving-cert" not found
Sep 18 19:56:56.986030 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:56:56.985838    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls podName:84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd nodeName:}" failed. No retries permitted until 2026-09-18 19:56:57.485815877 +0000 UTC m=+1108.448548596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls") pod "isvc-secondary-81daee-predictor-5b59c775bf-rbtsw" (UID: "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd") : secret "isvc-secondary-81daee-predictor-serving-cert" not found
Sep 18 19:56:56.986134 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.986112    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kserve-provision-location\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.986396 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.986377    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-cabundle-cert\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.986432 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.986392    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:56.994226 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:56.994206    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6lh\" (UniqueName: \"kubernetes.io/projected/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-api-access-xk6lh\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:57.112707 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:57.112633    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:56:57.491052 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:57.491012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:57.493359 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:57.493342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls\") pod \"isvc-secondary-81daee-predictor-5b59c775bf-rbtsw\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") " pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:57.730319 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:57.730279    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:56:58.004574 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:58.004457    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"]
Sep 18 19:56:58.007426 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:56:58.007399    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e16c1d_53db_4f9e_bf30_f1e93d4ed9cd.slice/crio-f2c233e12e806a1467c3d1ad7d047c411a431e62f2589a83a0fdc7892d586031 WatchSource:0}: Error finding container f2c233e12e806a1467c3d1ad7d047c411a431e62f2589a83a0fdc7892d586031: Status 404 returned error can't find the container with id f2c233e12e806a1467c3d1ad7d047c411a431e62f2589a83a0fdc7892d586031
Sep 18 19:56:58.573698 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:58.573659    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw" event={"ID":"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd","Type":"ContainerStarted","Data":"40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a"}
Sep 18 19:56:58.573874 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:56:58.573719    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw" event={"ID":"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd","Type":"ContainerStarted","Data":"f2c233e12e806a1467c3d1ad7d047c411a431e62f2589a83a0fdc7892d586031"}
Sep 18 19:57:02.591461 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:02.591431    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd/storage-initializer/0.log"
Sep 18 19:57:02.591863 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:02.591471    2559 generic.go:356] "Generic (PLEG): container finished" podID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" containerID="40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a" exitCode=1
Sep 18 19:57:02.591863 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:02.591506    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw" event={"ID":"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd","Type":"ContainerDied","Data":"40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a"}
Sep 18 19:57:03.597002 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:03.596954    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd/storage-initializer/0.log"
Sep 18 19:57:03.597377 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:03.597071    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw" event={"ID":"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd","Type":"ContainerStarted","Data":"631d3a01bae6300f3b8fac4d6631ac49bd1b2413ff807e25c481369fe482dfba"}
Sep 18 19:57:06.610585 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:06.610560    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd/storage-initializer/1.log"
Sep 18 19:57:06.611008 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:06.610993    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd/storage-initializer/0.log"
Sep 18 19:57:06.611073 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:06.611027    2559 generic.go:356] "Generic (PLEG): container finished" podID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" containerID="631d3a01bae6300f3b8fac4d6631ac49bd1b2413ff807e25c481369fe482dfba" exitCode=1
Sep 18 19:57:06.611073 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:06.611064    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw" event={"ID":"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd","Type":"ContainerDied","Data":"631d3a01bae6300f3b8fac4d6631ac49bd1b2413ff807e25c481369fe482dfba"}
Sep 18 19:57:06.611170 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:06.611122    2559 scope.go:117] "RemoveContainer" containerID="40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a"
Sep 18 19:57:06.611496 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:06.611479    2559 scope.go:117] "RemoveContainer" containerID="40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a"
Sep 18 19:57:06.623111 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:06.623081    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_kserve-ci-e2e-test(84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd)\"" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw" podUID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd"
Sep 18 19:57:06.623340 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:06.623312    2559 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_kserve-ci-e2e-test_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd_0 in pod sandbox f2c233e12e806a1467c3d1ad7d047c411a431e62f2589a83a0fdc7892d586031 from index: no such id: '40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a'" containerID="40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a"
Sep 18 19:57:06.623439 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:06.623351    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_kserve-ci-e2e-test_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd_0 in pod sandbox f2c233e12e806a1467c3d1ad7d047c411a431e62f2589a83a0fdc7892d586031 from index: no such id: '40f8f5bdcc03e7a44d79469650143b4f4661bbe044caff3393436c7a9b4e8a3a'"
Sep 18 19:57:07.617230 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:07.617204    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd/storage-initializer/1.log"
Sep 18 19:57:14.914801 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:14.914770    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"]
Sep 18 19:57:14.915423 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:14.915083    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" containerID="cri-o://0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07" gracePeriod=30
Sep 18 19:57:14.915423 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:14.915117    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kube-rbac-proxy" containerID="cri-o://fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d" gracePeriod=30
Sep 18 19:57:14.947347 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:14.947313    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"]
Sep 18 19:57:14.950740 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:14.950714    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:15.017437 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.017404    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:15.078343 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.078323    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd/storage-initializer/1.log"
Sep 18 19:57:15.080578 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.080553    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:57:15.130563 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.130528    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kserve-provision-location\") pod \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") "
Sep 18 19:57:15.130704 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.130588    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-cabundle-cert\") pod \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") "
Sep 18 19:57:15.130704 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.130618    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-rbac-proxy-sar-config\") pod \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") "
Sep 18 19:57:15.130704 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.130673    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls\") pod \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") "
Sep 18 19:57:15.130704 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.130702    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk6lh\" (UniqueName: \"kubernetes.io/projected/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-api-access-xk6lh\") pod \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\" (UID: \"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd\") "
Sep 18 19:57:15.130926 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.130884    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" (UID: "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:15.131036 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.131013    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" (UID: "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:15.131036 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.131022    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" (UID: "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:15.132825 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.132798    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-api-access-xk6lh" (OuterVolumeSpecName: "kube-api-access-xk6lh") pod "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" (UID: "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd"). InnerVolumeSpecName "kube-api-access-xk6lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:15.132932 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.132892    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" (UID: "84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:15.214690 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.214642    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Sep 18 19:57:15.219849 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.219823    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Sep 18 19:57:15.231832 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.231810    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:15.231895 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.231835    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:15.231895 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.231846    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:15.231895 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.231856    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:15.231895 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.231864    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6lh\" (UniqueName: \"kubernetes.io/projected/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd-kube-api-access-xk6lh\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:15.646047 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.646020    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-81daee-predictor-5b59c775bf-rbtsw_84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd/storage-initializer/1.log"
Sep 18 19:57:15.646244 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.646130    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"
Sep 18 19:57:15.646244 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.646127    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw" event={"ID":"84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd","Type":"ContainerDied","Data":"f2c233e12e806a1467c3d1ad7d047c411a431e62f2589a83a0fdc7892d586031"}
Sep 18 19:57:15.646244 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.646194    2559 scope.go:117] "RemoveContainer" containerID="631d3a01bae6300f3b8fac4d6631ac49bd1b2413ff807e25c481369fe482dfba"
Sep 18 19:57:15.649232 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.649210    2559 generic.go:356] "Generic (PLEG): container finished" podID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerID="fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d" exitCode=2
Sep 18 19:57:15.649342 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.649243    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" event={"ID":"da54697d-7fcb-44ad-a513-7e4a13b26c58","Type":"ContainerDied","Data":"fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d"}
Sep 18 19:57:15.678559 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.678529    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"]
Sep 18 19:57:15.682389 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:15.682364    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-81daee-predictor-5b59c775bf-rbtsw"]
Sep 18 19:57:17.469906 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:17.469870    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" path="/var/lib/kubelet/pods/84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd/volumes"
Sep 18 19:57:19.554691 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.554668    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:57:19.666451 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.666360    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd726\" (UniqueName: \"kubernetes.io/projected/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-api-access-rd726\") pod \"da54697d-7fcb-44ad-a513-7e4a13b26c58\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") "
Sep 18 19:57:19.666451 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.666407    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-cabundle-cert\") pod \"da54697d-7fcb-44ad-a513-7e4a13b26c58\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") "
Sep 18 19:57:19.666672 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.666476    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-rbac-proxy-sar-config\") pod \"da54697d-7fcb-44ad-a513-7e4a13b26c58\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") "
Sep 18 19:57:19.666672 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.666504    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da54697d-7fcb-44ad-a513-7e4a13b26c58-kserve-provision-location\") pod \"da54697d-7fcb-44ad-a513-7e4a13b26c58\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") "
Sep 18 19:57:19.666672 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.666577    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls\") pod \"da54697d-7fcb-44ad-a513-7e4a13b26c58\" (UID: \"da54697d-7fcb-44ad-a513-7e4a13b26c58\") "
Sep 18 19:57:19.666855 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.666830    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da54697d-7fcb-44ad-a513-7e4a13b26c58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "da54697d-7fcb-44ad-a513-7e4a13b26c58" (UID: "da54697d-7fcb-44ad-a513-7e4a13b26c58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:19.666910 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.666868    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "da54697d-7fcb-44ad-a513-7e4a13b26c58" (UID: "da54697d-7fcb-44ad-a513-7e4a13b26c58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:19.666910 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.666876    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "da54697d-7fcb-44ad-a513-7e4a13b26c58" (UID: "da54697d-7fcb-44ad-a513-7e4a13b26c58"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:19.667524 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.667498    2559 generic.go:356] "Generic (PLEG): container finished" podID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerID="0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07" exitCode=0
Sep 18 19:57:19.667656 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.667565    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"
Sep 18 19:57:19.667656 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.667602    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" event={"ID":"da54697d-7fcb-44ad-a513-7e4a13b26c58","Type":"ContainerDied","Data":"0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07"}
Sep 18 19:57:19.667771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.667668    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89" event={"ID":"da54697d-7fcb-44ad-a513-7e4a13b26c58","Type":"ContainerDied","Data":"817406f8652e0c82323f776d71650979e838f4462479fc8b05c5f54708df522b"}
Sep 18 19:57:19.667771 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.667694    2559 scope.go:117] "RemoveContainer" containerID="fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d"
Sep 18 19:57:19.668869 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.668749    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-api-access-rd726" (OuterVolumeSpecName: "kube-api-access-rd726") pod "da54697d-7fcb-44ad-a513-7e4a13b26c58" (UID: "da54697d-7fcb-44ad-a513-7e4a13b26c58"). InnerVolumeSpecName "kube-api-access-rd726". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:19.668947 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.668882    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "da54697d-7fcb-44ad-a513-7e4a13b26c58" (UID: "da54697d-7fcb-44ad-a513-7e4a13b26c58"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:19.681371 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.681353    2559 scope.go:117] "RemoveContainer" containerID="0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07"
Sep 18 19:57:19.688160 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.688145    2559 scope.go:117] "RemoveContainer" containerID="0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864"
Sep 18 19:57:19.694757 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.694732    2559 scope.go:117] "RemoveContainer" containerID="fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d"
Sep 18 19:57:19.695036 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:19.695012    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d\": container with ID starting with fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d not found: ID does not exist" containerID="fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d"
Sep 18 19:57:19.695087 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.695050    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d"} err="failed to get container status \"fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d\": rpc error: code = NotFound desc = could not find container \"fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d\": container with ID starting with fe0b34457a249a5c9fe8f8b6ab054e6a09431722451658c753768ebafa2e805d not found: ID does not exist"
Sep 18 19:57:19.695087 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.695074    2559 scope.go:117] "RemoveContainer" containerID="0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07"
Sep 18 19:57:19.695310 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:19.695292    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07\": container with ID starting with 0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07 not found: ID does not exist" containerID="0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07"
Sep 18 19:57:19.695378 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.695322    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07"} err="failed to get container status \"0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07\": rpc error: code = NotFound desc = could not find container \"0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07\": container with ID starting with 0a3a45e5d0f8e59e85ed57223e25bf7e2db579e03ca5b07f6d83f25ce3017f07 not found: ID does not exist"
Sep 18 19:57:19.695378 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.695346    2559 scope.go:117] "RemoveContainer" containerID="0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864"
Sep 18 19:57:19.695598 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:19.695584    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864\": container with ID starting with 0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864 not found: ID does not exist" containerID="0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864"
Sep 18 19:57:19.695646 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.695605    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864"} err="failed to get container status \"0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864\": rpc error: code = NotFound desc = could not find container \"0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864\": container with ID starting with 0ef8ffc9795b0eedd35d055286009c01e0bdd7fd756ec8c3ca2078d881ad6864 not found: ID does not exist"
Sep 18 19:57:19.767237 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.767206    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:19.767237 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.767231    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/da54697d-7fcb-44ad-a513-7e4a13b26c58-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:19.767377 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.767245    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/da54697d-7fcb-44ad-a513-7e4a13b26c58-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:19.767377 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.767255    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd726\" (UniqueName: \"kubernetes.io/projected/da54697d-7fcb-44ad-a513-7e4a13b26c58-kube-api-access-rd726\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:19.767377 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.767263    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/da54697d-7fcb-44ad-a513-7e4a13b26c58-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:19.991509 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.991480    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"]
Sep 18 19:57:19.997330 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:19.997305    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-81daee-predictor-865f44d78-xlh89"]
Sep 18 19:57:20.077542 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.077515    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:20.111236 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111202    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"]
Sep 18 19:57:20.111494 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111481    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="storage-initializer"
Sep 18 19:57:20.111549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111496    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="storage-initializer"
Sep 18 19:57:20.111549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111506    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" containerName="storage-initializer"
Sep 18 19:57:20.111549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111512    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" containerName="storage-initializer"
Sep 18 19:57:20.111549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111519    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" containerName="storage-initializer"
Sep 18 19:57:20.111549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111525    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" containerName="storage-initializer"
Sep 18 19:57:20.111549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111535    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container"
Sep 18 19:57:20.111549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111541    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container"
Sep 18 19:57:20.111549 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111549    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kube-rbac-proxy"
Sep 18 19:57:20.111860 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111554    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kube-rbac-proxy"
Sep 18 19:57:20.111860 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111598    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kube-rbac-proxy"
Sep 18 19:57:20.111860 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111605    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" containerName="storage-initializer"
Sep 18 19:57:20.111860 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111613    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="84e16c1d-53db-4f9e-bf30-f1e93d4ed9cd" containerName="storage-initializer"
Sep 18 19:57:20.111860 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.111619    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" containerName="kserve-container"
Sep 18 19:57:20.116035 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.116017    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.118183 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.118125    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c46f26-predictor-serving-cert\""
Sep 18 19:57:20.118183 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.118141    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 19:57:20.118365 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.118141    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-c46f26-kube-rbac-proxy-sar-config\""
Sep 18 19:57:20.118365 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.118227    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-c46f26\""
Sep 18 19:57:20.118365 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.118197    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-c46f26-dockercfg-vt6w4\""
Sep 18 19:57:20.123727 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.123707    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"]
Sep 18 19:57:20.170899 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.170868    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9f0f324-d328-4c4b-9646-b3011589817c-kserve-provision-location\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.171070 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.170954    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-cabundle-cert\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.171070 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.170992    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.171070 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.171025    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49b6\" (UniqueName: \"kubernetes.io/projected/d9f0f324-d328-4c4b-9646-b3011589817c-kube-api-access-z49b6\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.171178 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.171086    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.271875 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.271810    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-cabundle-cert\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.271875 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.271845    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.271875 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.271870    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49b6\" (UniqueName: \"kubernetes.io/projected/d9f0f324-d328-4c4b-9646-b3011589817c-kube-api-access-z49b6\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.272168 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.271893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.272168 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.271921    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9f0f324-d328-4c4b-9646-b3011589817c-kserve-provision-location\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.272168 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:20.272001    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-serving-cert: secret "isvc-init-fail-c46f26-predictor-serving-cert" not found
Sep 18 19:57:20.272168 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:20.272085    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls podName:d9f0f324-d328-4c4b-9646-b3011589817c nodeName:}" failed. No retries permitted until 2026-09-18 19:57:20.772064541 +0000 UTC m=+1131.734797259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls") pod "isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" (UID: "d9f0f324-d328-4c4b-9646-b3011589817c") : secret "isvc-init-fail-c46f26-predictor-serving-cert" not found
Sep 18 19:57:20.272544 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.272522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.272630 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.272547    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9f0f324-d328-4c4b-9646-b3011589817c-kserve-provision-location\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.272630 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.272527    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-cabundle-cert\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.279734 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.279716    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49b6\" (UniqueName: \"kubernetes.io/projected/d9f0f324-d328-4c4b-9646-b3011589817c-kube-api-access-z49b6\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.776015 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.775975    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:20.776404 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:20.776148    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-serving-cert: secret "isvc-init-fail-c46f26-predictor-serving-cert" not found
Sep 18 19:57:20.776404 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:20.776211    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls podName:d9f0f324-d328-4c4b-9646-b3011589817c nodeName:}" failed. No retries permitted until 2026-09-18 19:57:21.776195959 +0000 UTC m=+1132.738928676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls") pod "isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" (UID: "d9f0f324-d328-4c4b-9646-b3011589817c") : secret "isvc-init-fail-c46f26-predictor-serving-cert" not found
Sep 18 19:57:20.938427 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:20.938382    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:21.469348 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:21.469314    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da54697d-7fcb-44ad-a513-7e4a13b26c58" path="/var/lib/kubelet/pods/da54697d-7fcb-44ad-a513-7e4a13b26c58/volumes"
Sep 18 19:57:21.785608 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:21.785527    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:21.787878 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:21.787858    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls\") pod \"isvc-init-fail-c46f26-predictor-688947fcf7-2qj57\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") " pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:21.925690 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:21.925654    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:22.197299 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:22.197261    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"]
Sep 18 19:57:22.199471 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:57:22.199442    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f0f324_d328_4c4b_9646_b3011589817c.slice/crio-3361397757d87f613faacaff5bcedf1c6f7d01c57b35491cbf0dc59a2c46796f WatchSource:0}: Error finding container 3361397757d87f613faacaff5bcedf1c6f7d01c57b35491cbf0dc59a2c46796f: Status 404 returned error can't find the container with id 3361397757d87f613faacaff5bcedf1c6f7d01c57b35491cbf0dc59a2c46796f
Sep 18 19:57:22.680522 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:22.680487    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" event={"ID":"d9f0f324-d328-4c4b-9646-b3011589817c","Type":"ContainerStarted","Data":"9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e"}
Sep 18 19:57:22.680693 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:22.680551    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" event={"ID":"d9f0f324-d328-4c4b-9646-b3011589817c","Type":"ContainerStarted","Data":"3361397757d87f613faacaff5bcedf1c6f7d01c57b35491cbf0dc59a2c46796f"}
Sep 18 19:57:25.693044 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:25.692951    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_d9f0f324-d328-4c4b-9646-b3011589817c/storage-initializer/0.log"
Sep 18 19:57:25.693044 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:25.693006    2559 generic.go:356] "Generic (PLEG): container finished" podID="d9f0f324-d328-4c4b-9646-b3011589817c" containerID="9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e" exitCode=1
Sep 18 19:57:25.693480 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:25.693077    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" event={"ID":"d9f0f324-d328-4c4b-9646-b3011589817c","Type":"ContainerDied","Data":"9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e"}
Sep 18 19:57:26.698944 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:26.698915    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_d9f0f324-d328-4c4b-9646-b3011589817c/storage-initializer/0.log"
Sep 18 19:57:26.699376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:26.699031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" event={"ID":"d9f0f324-d328-4c4b-9646-b3011589817c","Type":"ContainerStarted","Data":"6a7db355cd5e1e3372a43b20bb60b46d971b447a5a3a4aa81384cccbb37d9e74"}
Sep 18 19:57:26.759560 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:26.759531    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-91b5b-predictor-598d8989f5-8jv89_192fe0a2-3850-4a97-bbf4-1c374a74db3c/kserve-container/0.log"
Sep 18 19:57:26.949746 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:26.949669    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:27.049547 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.049505    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:27.064168 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.064140    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"]
Sep 18 19:57:27.064372 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.064350    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerName="kserve-container" containerID="cri-o://a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4" gracePeriod=30
Sep 18 19:57:27.064435 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.064389    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerName="kube-rbac-proxy" containerID="cri-o://0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3" gracePeriod=30
Sep 18 19:57:27.072765 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.072736    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"]
Sep 18 19:57:27.076193 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.076171    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.078226 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.078209    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-1efdd-predictor-serving-cert\""
Sep 18 19:57:27.078441 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.078423    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-1efdd-kube-rbac-proxy-sar-config\""
Sep 18 19:57:27.078498 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.078474    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 19:57:27.094368 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.094324    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"]
Sep 18 19:57:27.148790 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.148762    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:27.231618 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.231584    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.231739 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.231631    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.231739 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.231707    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x65vx\" (UniqueName: \"kubernetes.io/projected/ba6a0aff-e782-488a-8949-474b9c442f33-kube-api-access-x65vx\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.231863 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.231742    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba6a0aff-e782-488a-8949-474b9c442f33-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.231863 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.231776    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.305319 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.305295    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:57:27.332346 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.332316    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x65vx\" (UniqueName: \"kubernetes.io/projected/ba6a0aff-e782-488a-8949-474b9c442f33-kube-api-access-x65vx\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.332502 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.332359    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba6a0aff-e782-488a-8949-474b9c442f33-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.332598 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.332547    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.332823 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.332695    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.332823 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.332728    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.332823 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.332769    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba6a0aff-e782-488a-8949-474b9c442f33-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.332823 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:27.332779    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-serving-cert: secret "isvc-sklearn-scale-raw-1efdd-predictor-serving-cert" not found
Sep 18 19:57:27.333095 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:27.332842    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls podName:ba6a0aff-e782-488a-8949-474b9c442f33 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:27.832821436 +0000 UTC m=+1138.795554168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls") pod "isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" (UID: "ba6a0aff-e782-488a-8949-474b9c442f33") : secret "isvc-sklearn-scale-raw-1efdd-predictor-serving-cert" not found
Sep 18 19:57:27.333276 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.333255    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.333346 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.333275    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-cabundle-cert\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.342514 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.342492    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x65vx\" (UniqueName: \"kubernetes.io/projected/ba6a0aff-e782-488a-8949-474b9c442f33-kube-api-access-x65vx\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.433324 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.433294    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59rw2\" (UniqueName: \"kubernetes.io/projected/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-api-access-59rw2\") pod \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") "
Sep 18 19:57:27.433514 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.433366    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls\") pod \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") "
Sep 18 19:57:27.433514 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.433391    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-rbac-proxy-sar-config\") pod \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\" (UID: \"192fe0a2-3850-4a97-bbf4-1c374a74db3c\") "
Sep 18 19:57:27.433792 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.433767    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "192fe0a2-3850-4a97-bbf4-1c374a74db3c" (UID: "192fe0a2-3850-4a97-bbf4-1c374a74db3c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:27.435452 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.435427    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "192fe0a2-3850-4a97-bbf4-1c374a74db3c" (UID: "192fe0a2-3850-4a97-bbf4-1c374a74db3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:27.435540 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.435473    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-api-access-59rw2" (OuterVolumeSpecName: "kube-api-access-59rw2") pod "192fe0a2-3850-4a97-bbf4-1c374a74db3c" (UID: "192fe0a2-3850-4a97-bbf4-1c374a74db3c"). InnerVolumeSpecName "kube-api-access-59rw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:27.534440 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.534354    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:27.534440 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.534383    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rw2\" (UniqueName: \"kubernetes.io/projected/192fe0a2-3850-4a97-bbf4-1c374a74db3c-kube-api-access-59rw2\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:27.534440 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.534393    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/192fe0a2-3850-4a97-bbf4-1c374a74db3c-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:27.661491 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.661459    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:27.704302 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.704265    2559 generic.go:356] "Generic (PLEG): container finished" podID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerID="0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3" exitCode=2
Sep 18 19:57:27.704302 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.704292    2559 generic.go:356] "Generic (PLEG): container finished" podID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerID="a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4" exitCode=2
Sep 18 19:57:27.704724 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.704323    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"
Sep 18 19:57:27.704724 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.704355    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" event={"ID":"192fe0a2-3850-4a97-bbf4-1c374a74db3c","Type":"ContainerDied","Data":"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3"}
Sep 18 19:57:27.704724 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.704425    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" event={"ID":"192fe0a2-3850-4a97-bbf4-1c374a74db3c","Type":"ContainerDied","Data":"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4"}
Sep 18 19:57:27.704724 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.704460    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89" event={"ID":"192fe0a2-3850-4a97-bbf4-1c374a74db3c","Type":"ContainerDied","Data":"1fc7f9d5e7968e45738fdef0812952bfe15ed28b51d2575d0f54aba7ae3d8fc1"}
Sep 18 19:57:27.704724 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.704488    2559 scope.go:117] "RemoveContainer" containerID="0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3"
Sep 18 19:57:27.712550 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.712534    2559 scope.go:117] "RemoveContainer" containerID="a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4"
Sep 18 19:57:27.720288 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.720264    2559 scope.go:117] "RemoveContainer" containerID="0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3"
Sep 18 19:57:27.720387 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.720365    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"]
Sep 18 19:57:27.720597 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:27.720579    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3\": container with ID starting with 0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3 not found: ID does not exist" containerID="0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3"
Sep 18 19:57:27.720639 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.720606    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3"} err="failed to get container status \"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3\": rpc error: code = NotFound desc = could not find container \"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3\": container with ID starting with 0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3 not found: ID does not exist"
Sep 18 19:57:27.720639 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.720623    2559 scope.go:117] "RemoveContainer" containerID="a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4"
Sep 18 19:57:27.720884 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:27.720868    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4\": container with ID starting with a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4 not found: ID does not exist" containerID="a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4"
Sep 18 19:57:27.720928 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.720891    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4"} err="failed to get container status \"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4\": rpc error: code = NotFound desc = could not find container \"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4\": container with ID starting with a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4 not found: ID does not exist"
Sep 18 19:57:27.720928 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.720906    2559 scope.go:117] "RemoveContainer" containerID="0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3"
Sep 18 19:57:27.721168 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.721150    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3"} err="failed to get container status \"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3\": rpc error: code = NotFound desc = could not find container \"0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3\": container with ID starting with 0120faecf02e844841f7a3f0f142612dc9bbf2029a453dbc2cb0a4f7cd366fd3 not found: ID does not exist"
Sep 18 19:57:27.721217 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.721169    2559 scope.go:117] "RemoveContainer" containerID="a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4"
Sep 18 19:57:27.721399 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.721383    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4"} err="failed to get container status \"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4\": rpc error: code = NotFound desc = could not find container \"a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4\": container with ID starting with a3ec8ade0e2e475c675b1b872a20a38a53e8dcb85d54d6e195a7e2cfa72c29f4 not found: ID does not exist"
Sep 18 19:57:27.723705 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.723687    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-91b5b-predictor-598d8989f5-8jv89"]
Sep 18 19:57:27.836565 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.836480    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:27.838859 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:27.838841    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls\") pod \"isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:28.044231 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:28.044193    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:28.314177 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:28.313857    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"]
Sep 18 19:57:28.316439 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:57:28.316413    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6a0aff_e782_488a_8949_474b9c442f33.slice/crio-6777444c3c389c33980af11e1345beb36da1eb505e7b48f9951bb9465744cf79 WatchSource:0}: Error finding container 6777444c3c389c33980af11e1345beb36da1eb505e7b48f9951bb9465744cf79: Status 404 returned error can't find the container with id 6777444c3c389c33980af11e1345beb36da1eb505e7b48f9951bb9465744cf79
Sep 18 19:57:28.710511 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:28.710476    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" event={"ID":"ba6a0aff-e782-488a-8949-474b9c442f33","Type":"ContainerStarted","Data":"60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4"}
Sep 18 19:57:28.710985 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:28.710541    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" event={"ID":"ba6a0aff-e782-488a-8949-474b9c442f33","Type":"ContainerStarted","Data":"6777444c3c389c33980af11e1345beb36da1eb505e7b48f9951bb9465744cf79"}
Sep 18 19:57:29.470158 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:29.470121    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" path="/var/lib/kubelet/pods/192fe0a2-3850-4a97-bbf4-1c374a74db3c/volumes"
Sep 18 19:57:29.715796 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:29.715770    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_d9f0f324-d328-4c4b-9646-b3011589817c/storage-initializer/1.log"
Sep 18 19:57:29.716235 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:29.716176    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_d9f0f324-d328-4c4b-9646-b3011589817c/storage-initializer/0.log"
Sep 18 19:57:29.716235 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:29.716210    2559 generic.go:356] "Generic (PLEG): container finished" podID="d9f0f324-d328-4c4b-9646-b3011589817c" containerID="6a7db355cd5e1e3372a43b20bb60b46d971b447a5a3a4aa81384cccbb37d9e74" exitCode=1
Sep 18 19:57:29.716311 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:29.716285    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" event={"ID":"d9f0f324-d328-4c4b-9646-b3011589817c","Type":"ContainerDied","Data":"6a7db355cd5e1e3372a43b20bb60b46d971b447a5a3a4aa81384cccbb37d9e74"}
Sep 18 19:57:29.716375 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:29.716345    2559 scope.go:117] "RemoveContainer" containerID="9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e"
Sep 18 19:57:29.716706 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:29.716686    2559 scope.go:117] "RemoveContainer" containerID="9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e"
Sep 18 19:57:29.726596 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:29.726572    2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_kserve-ci-e2e-test(d9f0f324-d328-4c4b-9646-b3011589817c)\"" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" podUID="d9f0f324-d328-4c4b-9646-b3011589817c"
Sep 18 19:57:29.726810 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:29.726781    2559 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_kserve-ci-e2e-test_d9f0f324-d328-4c4b-9646-b3011589817c_0 in pod sandbox 3361397757d87f613faacaff5bcedf1c6f7d01c57b35491cbf0dc59a2c46796f from index: no such id: '9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e'" containerID="9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e"
Sep 18 19:57:29.726869 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:29.726818    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_kserve-ci-e2e-test_d9f0f324-d328-4c4b-9646-b3011589817c_0 in pod sandbox 3361397757d87f613faacaff5bcedf1c6f7d01c57b35491cbf0dc59a2c46796f from index: no such id: '9ce7532cd5a04736f7868ab33735f0d6d85428635eebde3e5784c519fe8ab14e'"
Sep 18 19:57:30.072015 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.071923    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"]
Sep 18 19:57:30.094939 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.094904    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:30.721734 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.721712    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_d9f0f324-d328-4c4b-9646-b3011589817c/storage-initializer/1.log"
Sep 18 19:57:30.845261 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.845243    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-c46f26-predictor-688947fcf7-2qj57_d9f0f324-d328-4c4b-9646-b3011589817c/storage-initializer/1.log"
Sep 18 19:57:30.848782 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.848765    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:30.962174 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.962145    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-cabundle-cert\") pod \"d9f0f324-d328-4c4b-9646-b3011589817c\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") "
Sep 18 19:57:30.962300 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.962193    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls\") pod \"d9f0f324-d328-4c4b-9646-b3011589817c\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") "
Sep 18 19:57:30.962300 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.962242    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9f0f324-d328-4c4b-9646-b3011589817c-kserve-provision-location\") pod \"d9f0f324-d328-4c4b-9646-b3011589817c\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") "
Sep 18 19:57:30.962300 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.962267    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z49b6\" (UniqueName: \"kubernetes.io/projected/d9f0f324-d328-4c4b-9646-b3011589817c-kube-api-access-z49b6\") pod \"d9f0f324-d328-4c4b-9646-b3011589817c\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") "
Sep 18 19:57:30.962461 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.962304    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-kube-rbac-proxy-sar-config\") pod \"d9f0f324-d328-4c4b-9646-b3011589817c\" (UID: \"d9f0f324-d328-4c4b-9646-b3011589817c\") "
Sep 18 19:57:30.962527 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.962500    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f0f324-d328-4c4b-9646-b3011589817c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d9f0f324-d328-4c4b-9646-b3011589817c" (UID: "d9f0f324-d328-4c4b-9646-b3011589817c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:57:30.962632 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.962536    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d9f0f324-d328-4c4b-9646-b3011589817c" (UID: "d9f0f324-d328-4c4b-9646-b3011589817c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:30.964232 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.964204    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d9f0f324-d328-4c4b-9646-b3011589817c" (UID: "d9f0f324-d328-4c4b-9646-b3011589817c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:57:30.964314 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.964263    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d9f0f324-d328-4c4b-9646-b3011589817c" (UID: "d9f0f324-d328-4c4b-9646-b3011589817c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:57:30.965653 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:30.965634    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f0f324-d328-4c4b-9646-b3011589817c-kube-api-access-z49b6" (OuterVolumeSpecName: "kube-api-access-z49b6") pod "d9f0f324-d328-4c4b-9646-b3011589817c" (UID: "d9f0f324-d328-4c4b-9646-b3011589817c"). InnerVolumeSpecName "kube-api-access-z49b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:57:31.063376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.063309    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:31.063376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.063333    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0f324-d328-4c4b-9646-b3011589817c-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:31.063376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.063344    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d9f0f324-d328-4c4b-9646-b3011589817c-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:31.063376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.063352    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d9f0f324-d328-4c4b-9646-b3011589817c-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:31.063376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.063360    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z49b6\" (UniqueName: \"kubernetes.io/projected/d9f0f324-d328-4c4b-9646-b3011589817c-kube-api-access-z49b6\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:57:31.725835 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.725750    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"
Sep 18 19:57:31.725835 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.725757    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57" event={"ID":"d9f0f324-d328-4c4b-9646-b3011589817c","Type":"ContainerDied","Data":"3361397757d87f613faacaff5bcedf1c6f7d01c57b35491cbf0dc59a2c46796f"}
Sep 18 19:57:31.725835 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.725829    2559 scope.go:117] "RemoveContainer" containerID="6a7db355cd5e1e3372a43b20bb60b46d971b447a5a3a4aa81384cccbb37d9e74"
Sep 18 19:57:31.776201 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.776174    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"]
Sep 18 19:57:31.777769 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:31.777743    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-c46f26-predictor-688947fcf7-2qj57"]
Sep 18 19:57:32.731046 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:32.731010    2559 generic.go:356] "Generic (PLEG): container finished" podID="ba6a0aff-e782-488a-8949-474b9c442f33" containerID="60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4" exitCode=0
Sep 18 19:57:32.731493 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:32.731084    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" event={"ID":"ba6a0aff-e782-488a-8949-474b9c442f33","Type":"ContainerDied","Data":"60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4"}
Sep 18 19:57:33.470126 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:33.470086    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9f0f324-d328-4c4b-9646-b3011589817c" path="/var/lib/kubelet/pods/d9f0f324-d328-4c4b-9646-b3011589817c/volumes"
Sep 18 19:57:33.738594 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:33.738515    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" event={"ID":"ba6a0aff-e782-488a-8949-474b9c442f33","Type":"ContainerStarted","Data":"d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512"}
Sep 18 19:57:33.738594 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:33.738573    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" event={"ID":"ba6a0aff-e782-488a-8949-474b9c442f33","Type":"ContainerStarted","Data":"972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee"}
Sep 18 19:57:33.739093 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:33.738844    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:33.739093 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:33.739011    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:33.740370 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:33.740343    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:57:33.756757 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:33.756714    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podStartSLOduration=2.926179406 podStartE2EDuration="6.756701485s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.830522079s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:33.755144151 +0000 UTC m=+1144.717876889" watchObservedRunningTime="2026-09-18 19:57:33.756701485 +0000 UTC m=+1144.719434222"
Sep 18 19:57:34.743087 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:34.743049    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:57:35.305533 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.305470    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:35.346497 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346466    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"]
Sep 18 19:57:35.346758 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346746    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f0f324-d328-4c4b-9646-b3011589817c" containerName="storage-initializer"
Sep 18 19:57:35.346818 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346760    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f0f324-d328-4c4b-9646-b3011589817c" containerName="storage-initializer"
Sep 18 19:57:35.346818 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346769    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerName="kube-rbac-proxy"
Sep 18 19:57:35.346818 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346775    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerName="kube-rbac-proxy"
Sep 18 19:57:35.346818 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346793    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerName="kserve-container"
Sep 18 19:57:35.346818 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346798    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerName="kserve-container"
Sep 18 19:57:35.347029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346838    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerName="kube-rbac-proxy"
Sep 18 19:57:35.347029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346846    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="192fe0a2-3850-4a97-bbf4-1c374a74db3c" containerName="kserve-container"
Sep 18 19:57:35.347029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346853    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f0f324-d328-4c4b-9646-b3011589817c" containerName="storage-initializer"
Sep 18 19:57:35.347029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346858    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f0f324-d328-4c4b-9646-b3011589817c" containerName="storage-initializer"
Sep 18 19:57:35.347029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346907    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f0f324-d328-4c4b-9646-b3011589817c" containerName="storage-initializer"
Sep 18 19:57:35.347029 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.346913    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f0f324-d328-4c4b-9646-b3011589817c" containerName="storage-initializer"
Sep 18 19:57:35.349624 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.349608    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.352151 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.352129    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fcbfb-predictor-serving-cert\""
Sep 18 19:57:35.352151 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.352140    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fcbfb-kube-rbac-proxy-sar-config\""
Sep 18 19:57:35.361512 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.361488    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"]
Sep 18 19:57:35.500475 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.500441    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.500475 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.500477    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-cabundle-cert\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.500664 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.500501    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.500664 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.500571    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/425bd21e-f06d-402f-8096-7af036a48170-kserve-provision-location\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.500664 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.500622    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29jk\" (UniqueName: \"kubernetes.io/projected/425bd21e-f06d-402f-8096-7af036a48170-kube-api-access-x29jk\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.601234 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.601154    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/425bd21e-f06d-402f-8096-7af036a48170-kserve-provision-location\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.601234 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.601204    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29jk\" (UniqueName: \"kubernetes.io/projected/425bd21e-f06d-402f-8096-7af036a48170-kube-api-access-x29jk\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.601234 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.601237    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.601572 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.601261    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-cabundle-cert\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.601572 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.601280    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.601572 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:35.601479    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-serving-cert: secret "raw-sklearn-fcbfb-predictor-serving-cert" not found
Sep 18 19:57:35.601572 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:57:35.601546    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls podName:425bd21e-f06d-402f-8096-7af036a48170 nodeName:}" failed. No retries permitted until 2026-09-18 19:57:36.101524211 +0000 UTC m=+1147.064256947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls") pod "raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" (UID: "425bd21e-f06d-402f-8096-7af036a48170") : secret "raw-sklearn-fcbfb-predictor-serving-cert" not found
Sep 18 19:57:35.601736 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.601562    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/425bd21e-f06d-402f-8096-7af036a48170-kserve-provision-location\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.601844 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.601825    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-cabundle-cert\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.601898 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.601882    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.609560 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.609536    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29jk\" (UniqueName: \"kubernetes.io/projected/425bd21e-f06d-402f-8096-7af036a48170-kube-api-access-x29jk\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:35.698127 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.698091    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:57:35.746542 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:35.746509    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:57:36.106245 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:36.106200    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:36.108655 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:36.108630    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls\") pod \"raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") " pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:36.259973 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:36.259923    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:36.557737 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:36.557708    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"]
Sep 18 19:57:36.560656 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:57:36.560628    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod425bd21e_f06d_402f_8096_7af036a48170.slice/crio-8edd7e165913d2eb60bc747a0704fc52dc61e8fda9bcf177087c5158fe068e63 WatchSource:0}: Error finding container 8edd7e165913d2eb60bc747a0704fc52dc61e8fda9bcf177087c5158fe068e63: Status 404 returned error can't find the container with id 8edd7e165913d2eb60bc747a0704fc52dc61e8fda9bcf177087c5158fe068e63
Sep 18 19:57:36.752057 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:36.752022    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" event={"ID":"425bd21e-f06d-402f-8096-7af036a48170","Type":"ContainerStarted","Data":"b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27"}
Sep 18 19:57:36.752398 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:36.752099    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" event={"ID":"425bd21e-f06d-402f-8096-7af036a48170","Type":"ContainerStarted","Data":"8edd7e165913d2eb60bc747a0704fc52dc61e8fda9bcf177087c5158fe068e63"}
Sep 18 19:57:40.752708 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:40.752682    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 19:57:40.753182 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:40.753155    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:57:40.767948 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:40.767924    2559 generic.go:356] "Generic (PLEG): container finished" podID="425bd21e-f06d-402f-8096-7af036a48170" containerID="b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27" exitCode=0
Sep 18 19:57:40.768069 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:40.767996    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" event={"ID":"425bd21e-f06d-402f-8096-7af036a48170","Type":"ContainerDied","Data":"b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27"}
Sep 18 19:57:41.773712 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:41.773678    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" event={"ID":"425bd21e-f06d-402f-8096-7af036a48170","Type":"ContainerStarted","Data":"d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95"}
Sep 18 19:57:41.774177 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:41.773749    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" event={"ID":"425bd21e-f06d-402f-8096-7af036a48170","Type":"ContainerStarted","Data":"7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb"}
Sep 18 19:57:41.774177 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:41.773988    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:41.792235 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:41.792189    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podStartSLOduration=3.040859494 podStartE2EDuration="6.792172714s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.75131322s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:57:41.790319813 +0000 UTC m=+1152.753052563" watchObservedRunningTime="2026-09-18 19:57:41.792172714 +0000 UTC m=+1152.754905452"
Sep 18 19:57:42.777376 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:42.777298    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:42.778598 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:42.778568    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:57:43.781017 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:43.780978    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:57:48.785503 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:48.785472    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:57:48.786004 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:48.785933    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:57:50.754145 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:50.754101    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:57:58.786328 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:57:58.786286    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:58:00.753723 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:00.753680    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:58:08.786542 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:08.786504    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:58:10.753666 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:10.753616    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:58:18.786561 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:18.786520    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:58:20.753370 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:20.753326    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:58:28.224240 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:28.224204    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:58:28.786711 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:28.786669    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:58:30.753485 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:30.753440    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:58:31.414667 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:31.414642    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:58:31.415333 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:31.415316    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 19:58:38.786391 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:38.786349    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:58:40.754163 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:40.754122    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:58:44.466141 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:44.466093    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:58:48.786276 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:48.786244    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:58:54.466476 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:54.466438    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:58:58.787045 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:58:58.787011    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:59:04.466845 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:04.466805    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:59:05.413043 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:05.413014    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"]
Sep 18 19:59:05.413280 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:05.413252    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" containerID="cri-o://7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb" gracePeriod=30
Sep 18 19:59:05.413392 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:05.413298    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kube-rbac-proxy" containerID="cri-o://d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95" gracePeriod=30
Sep 18 19:59:05.425566 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:05.425545    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:06.086502 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:06.086472    2559 generic.go:356] "Generic (PLEG): container finished" podID="425bd21e-f06d-402f-8096-7af036a48170" containerID="d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95" exitCode=2
Sep 18 19:59:06.086849 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:06.086539    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" event={"ID":"425bd21e-f06d-402f-8096-7af036a48170","Type":"ContainerDied","Data":"d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95"}
Sep 18 19:59:08.781608 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:08.781566    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 18 19:59:08.786062 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:08.786022    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 18 19:59:09.246785 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.246763    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:59:09.374847 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.374782    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls\") pod \"425bd21e-f06d-402f-8096-7af036a48170\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") "
Sep 18 19:59:09.374847 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.374817    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x29jk\" (UniqueName: \"kubernetes.io/projected/425bd21e-f06d-402f-8096-7af036a48170-kube-api-access-x29jk\") pod \"425bd21e-f06d-402f-8096-7af036a48170\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") "
Sep 18 19:59:09.374847 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.374844    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/425bd21e-f06d-402f-8096-7af036a48170-kserve-provision-location\") pod \"425bd21e-f06d-402f-8096-7af036a48170\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") "
Sep 18 19:59:09.375179 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.374871    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-cabundle-cert\") pod \"425bd21e-f06d-402f-8096-7af036a48170\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") "
Sep 18 19:59:09.375179 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.374949    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-kube-rbac-proxy-sar-config\") pod \"425bd21e-f06d-402f-8096-7af036a48170\" (UID: \"425bd21e-f06d-402f-8096-7af036a48170\") "
Sep 18 19:59:09.375266 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.375170    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425bd21e-f06d-402f-8096-7af036a48170-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "425bd21e-f06d-402f-8096-7af036a48170" (UID: "425bd21e-f06d-402f-8096-7af036a48170"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 19:59:09.375302 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.375268    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "425bd21e-f06d-402f-8096-7af036a48170" (UID: "425bd21e-f06d-402f-8096-7af036a48170"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:09.375367 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.375349    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "425bd21e-f06d-402f-8096-7af036a48170" (UID: "425bd21e-f06d-402f-8096-7af036a48170"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 19:59:09.376827 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.376812    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425bd21e-f06d-402f-8096-7af036a48170-kube-api-access-x29jk" (OuterVolumeSpecName: "kube-api-access-x29jk") pod "425bd21e-f06d-402f-8096-7af036a48170" (UID: "425bd21e-f06d-402f-8096-7af036a48170"). InnerVolumeSpecName "kube-api-access-x29jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 19:59:09.376907 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.376893    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "425bd21e-f06d-402f-8096-7af036a48170" (UID: "425bd21e-f06d-402f-8096-7af036a48170"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 19:59:09.475618 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.475591    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/425bd21e-f06d-402f-8096-7af036a48170-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:59:09.475732 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.475623    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x29jk\" (UniqueName: \"kubernetes.io/projected/425bd21e-f06d-402f-8096-7af036a48170-kube-api-access-x29jk\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:59:09.475732 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.475637    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/425bd21e-f06d-402f-8096-7af036a48170-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:59:09.475732 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.475649    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:59:09.475732 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:09.475661    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/425bd21e-f06d-402f-8096-7af036a48170-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 19:59:10.103697 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.103670    2559 generic.go:356] "Generic (PLEG): container finished" podID="425bd21e-f06d-402f-8096-7af036a48170" containerID="7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb" exitCode=0
Sep 18 19:59:10.104092 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.103729    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" event={"ID":"425bd21e-f06d-402f-8096-7af036a48170","Type":"ContainerDied","Data":"7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb"}
Sep 18 19:59:10.104092 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.103743    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"
Sep 18 19:59:10.104092 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.103779    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk" event={"ID":"425bd21e-f06d-402f-8096-7af036a48170","Type":"ContainerDied","Data":"8edd7e165913d2eb60bc747a0704fc52dc61e8fda9bcf177087c5158fe068e63"}
Sep 18 19:59:10.104092 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.103805    2559 scope.go:117] "RemoveContainer" containerID="d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95"
Sep 18 19:59:10.112042 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.112027    2559 scope.go:117] "RemoveContainer" containerID="7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb"
Sep 18 19:59:10.119134 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.118989    2559 scope.go:117] "RemoveContainer" containerID="b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27"
Sep 18 19:59:10.120466 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.120443    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"]
Sep 18 19:59:10.123662 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.123644    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fcbfb-predictor-58ccc8ff55-jlbtk"]
Sep 18 19:59:10.126444 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.126427    2559 scope.go:117] "RemoveContainer" containerID="d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95"
Sep 18 19:59:10.126718 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:59:10.126699    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95\": container with ID starting with d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95 not found: ID does not exist" containerID="d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95"
Sep 18 19:59:10.126764 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.126726    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95"} err="failed to get container status \"d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95\": rpc error: code = NotFound desc = could not find container \"d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95\": container with ID starting with d6bf5a2545f28cf054b89816e41f57725fb85a20592e9534617c36bca1648e95 not found: ID does not exist"
Sep 18 19:59:10.126764 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.126742    2559 scope.go:117] "RemoveContainer" containerID="7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb"
Sep 18 19:59:10.127002 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:59:10.126983    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb\": container with ID starting with 7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb not found: ID does not exist" containerID="7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb"
Sep 18 19:59:10.127084 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.127013    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb"} err="failed to get container status \"7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb\": rpc error: code = NotFound desc = could not find container \"7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb\": container with ID starting with 7a7baf226d3b6cbba2853ec33eee4edee77a0fefabe6b2c9bbdca4d35b2795fb not found: ID does not exist"
Sep 18 19:59:10.127084 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.127035    2559 scope.go:117] "RemoveContainer" containerID="b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27"
Sep 18 19:59:10.127287 ip-10-0-141-190 kubenswrapper[2559]: E0918 19:59:10.127269    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27\": container with ID starting with b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27 not found: ID does not exist" containerID="b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27"
Sep 18 19:59:10.127328 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.127291    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27"} err="failed to get container status \"b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27\": rpc error: code = NotFound desc = could not find container \"b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27\": container with ID starting with b9b174a372aabd656f8dd909f6910eb618219297d5429c2cea2b8e0c706a7f27 not found: ID does not exist"
Sep 18 19:59:10.529788 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.529754    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:10.584602 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584576    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"]
Sep 18 19:59:10.584833 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584821    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kube-rbac-proxy"
Sep 18 19:59:10.584874 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584835    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kube-rbac-proxy"
Sep 18 19:59:10.584874 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584851    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="storage-initializer"
Sep 18 19:59:10.584874 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584856    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="storage-initializer"
Sep 18 19:59:10.584874 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584865    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container"
Sep 18 19:59:10.584874 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584870    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container"
Sep 18 19:59:10.585073 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584912    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kserve-container"
Sep 18 19:59:10.585073 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.584921    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="425bd21e-f06d-402f-8096-7af036a48170" containerName="kube-rbac-proxy"
Sep 18 19:59:10.589570 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.589556    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.591459 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.591441    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-aa327-kube-rbac-proxy-sar-config\""
Sep 18 19:59:10.591547 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.591477    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-aa327-predictor-serving-cert\""
Sep 18 19:59:10.596816 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.596796    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"]
Sep 18 19:59:10.654258 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.654236    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 19:59:10.681307 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.681284    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c05583-2a24-492b-8d80-f9022c3804ff-proxy-tls\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.681379 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.681310    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-cabundle-cert\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.681379 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.681337    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986l7\" (UniqueName: \"kubernetes.io/projected/19c05583-2a24-492b-8d80-f9022c3804ff-kube-api-access-986l7\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.681379 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.681359    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c05583-2a24-492b-8d80-f9022c3804ff-kserve-provision-location\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.681490 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.681393    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.782631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.782579    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986l7\" (UniqueName: \"kubernetes.io/projected/19c05583-2a24-492b-8d80-f9022c3804ff-kube-api-access-986l7\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.782631 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.782611    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c05583-2a24-492b-8d80-f9022c3804ff-kserve-provision-location\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.782769 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.782635    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.782769 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.782679    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c05583-2a24-492b-8d80-f9022c3804ff-proxy-tls\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.782769 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.782698    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-cabundle-cert\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.782997 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.782951    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c05583-2a24-492b-8d80-f9022c3804ff-kserve-provision-location\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.783339 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.783319    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.783406 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.783318    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-cabundle-cert\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.785032 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.785011    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c05583-2a24-492b-8d80-f9022c3804ff-proxy-tls\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.789782 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.789754    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-986l7\" (UniqueName: \"kubernetes.io/projected/19c05583-2a24-492b-8d80-f9022c3804ff-kube-api-access-986l7\") pod \"raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:10.899598 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:10.899572    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:11.165956 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:11.165787    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"]
Sep 18 19:59:11.168565 ip-10-0-141-190 kubenswrapper[2559]: W0918 19:59:11.168542    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c05583_2a24_492b_8d80_f9022c3804ff.slice/crio-d5ef9eb25785c2d1d9d821bd5c249a9ca96ebc1b0ab271319b516646fe813d62 WatchSource:0}: Error finding container d5ef9eb25785c2d1d9d821bd5c249a9ca96ebc1b0ab271319b516646fe813d62: Status 404 returned error can't find the container with id d5ef9eb25785c2d1d9d821bd5c249a9ca96ebc1b0ab271319b516646fe813d62
Sep 18 19:59:11.470128 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:11.470099    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="425bd21e-f06d-402f-8096-7af036a48170" path="/var/lib/kubelet/pods/425bd21e-f06d-402f-8096-7af036a48170/volumes"
Sep 18 19:59:12.112482 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:12.112452    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" event={"ID":"19c05583-2a24-492b-8d80-f9022c3804ff","Type":"ContainerStarted","Data":"a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea"}
Sep 18 19:59:12.112630 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:12.112511    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" event={"ID":"19c05583-2a24-492b-8d80-f9022c3804ff","Type":"ContainerStarted","Data":"d5ef9eb25785c2d1d9d821bd5c249a9ca96ebc1b0ab271319b516646fe813d62"}
Sep 18 19:59:14.466428 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:14.466383    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:59:15.124606 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:15.124575    2559 generic.go:356] "Generic (PLEG): container finished" podID="19c05583-2a24-492b-8d80-f9022c3804ff" containerID="a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea" exitCode=0
Sep 18 19:59:15.124757 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:15.124627    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" event={"ID":"19c05583-2a24-492b-8d80-f9022c3804ff","Type":"ContainerDied","Data":"a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea"}
Sep 18 19:59:16.130073 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:16.130031    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" event={"ID":"19c05583-2a24-492b-8d80-f9022c3804ff","Type":"ContainerStarted","Data":"c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a"}
Sep 18 19:59:16.130477 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:16.130124    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" event={"ID":"19c05583-2a24-492b-8d80-f9022c3804ff","Type":"ContainerStarted","Data":"12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf"}
Sep 18 19:59:16.130477 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:16.130304    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:16.150869 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:16.150824    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podStartSLOduration=2.436828159 podStartE2EDuration="6.150812541s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.713984382s" isStatefulPod=false podCreationTimestamp="2026-09-18 19:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 19:59:16.149635149 +0000 UTC m=+1247.112367886" watchObservedRunningTime="2026-09-18 19:59:16.150812541 +0000 UTC m=+1247.113545280"
Sep 18 19:59:17.134200 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:17.134169    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:17.135750 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:17.135718    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:59:18.137860 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:18.137815    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:59:23.142855 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:23.142825    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 19:59:23.143332 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:23.143305    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:59:24.466432 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:24.466387    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:59:33.144261 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:33.144223    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:59:34.466007 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:34.465954    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:59:43.144072 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:43.144030    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:59:44.466503 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:44.466466    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:59:53.144070 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:53.144024    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 19:59:54.466182 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:54.466140    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 19:59:59.589994 ip-10-0-141-190 kubenswrapper[2559]: I0918 19:59:59.589943    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:02.466137 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:02.466107    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 20:00:03.143459 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:03.143416    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 20:00:07.100270 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:07.100228    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"]
Sep 18 20:00:07.100714 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:07.100581    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" containerID="cri-o://972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee" gracePeriod=30
Sep 18 20:00:07.100714 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:07.100646    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kube-rbac-proxy" containerID="cri-o://d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512" gracePeriod=30
Sep 18 20:00:07.145835 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:07.145790    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:07.326372 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:07.326336    2559 generic.go:356] "Generic (PLEG): container finished" podID="ba6a0aff-e782-488a-8949-474b9c442f33" containerID="d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512" exitCode=2
Sep 18 20:00:07.326551 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:07.326408    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" event={"ID":"ba6a0aff-e782-488a-8949-474b9c442f33","Type":"ContainerDied","Data":"d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512"}
Sep 18 20:00:10.746796 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:10.746754    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 20:00:12.285393 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.285355    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:12.318647 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.318620    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"]
Sep 18 20:00:12.322081 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.322059    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.324165 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.324144    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 18 20:00:12.324271 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.324193    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 18 20:00:12.331940 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.331921    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"]
Sep 18 20:00:12.354223 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.354192    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:12.395831 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.395808    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"]
Sep 18 20:00:12.398936 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.398922    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.400709 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.400691    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 18 20:00:12.408558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.408540    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"]
Sep 18 20:00:12.424063 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.424037    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.424146 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.424070    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsbz\" (UniqueName: \"kubernetes.io/projected/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-api-access-vnsbz\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.424146 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.424098    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.424223 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.424181    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.424258 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.424233    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.465929 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.465899    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Sep 18 20:00:12.524764 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.524739    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.524865 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.524773    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.524865 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.524801    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsbz\" (UniqueName: \"kubernetes.io/projected/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-api-access-vnsbz\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.524865 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.524854    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.525008 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.524897    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.525008 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.524921    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.525008 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.524982    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctsmw\" (UniqueName: \"kubernetes.io/projected/16afa416-1cf6-45d2-b374-54515798f714-kube-api-access-ctsmw\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.525166 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.525012    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.525166 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:12.525031    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-v2-predictor-serving-cert: secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 18 20:00:12.525277 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.525167    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.525277 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:12.525190    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls podName:261da5a3-9952-47a6-a373-3bdb58c7c8e1 nodeName:}" failed. No retries permitted until 2026-09-18 20:00:13.025167107 +0000 UTC m=+1303.987899839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls") pod "isvc-canary-create-v2-predictor-579784d7d7-wfkvd" (UID: "261da5a3-9952-47a6-a373-3bdb58c7c8e1") : secret "isvc-canary-create-v2-predictor-serving-cert" not found
Sep 18 20:00:12.525277 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.525234    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16afa416-1cf6-45d2-b374-54515798f714-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.525277 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.525264    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.525524 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.525502    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.525687 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.525666    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.534942 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.534921    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsbz\" (UniqueName: \"kubernetes.io/projected/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-api-access-vnsbz\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:12.625949 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.625893    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.625949 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.625947    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.626111 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.625989    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsmw\" (UniqueName: \"kubernetes.io/projected/16afa416-1cf6-45d2-b374-54515798f714-kube-api-access-ctsmw\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.626111 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.626021    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16afa416-1cf6-45d2-b374-54515798f714-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.626111 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.626047    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.626111 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:12.626061    2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:00:12.626274 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:12.626148    2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls podName:16afa416-1cf6-45d2-b374-54515798f714 nodeName:}" failed. No retries permitted until 2026-09-18 20:00:13.126126792 +0000 UTC m=+1304.088859521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls") pod "isvc-canary-create-predictor-68984b9498-l98dp" (UID: "16afa416-1cf6-45d2-b374-54515798f714") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 18 20:00:12.626431 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.626409    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16afa416-1cf6-45d2-b374-54515798f714-kserve-provision-location\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.626586 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.626568    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-cabundle-cert\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.626670 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.626655    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.633812 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.633793    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsmw\" (UniqueName: \"kubernetes.io/projected/16afa416-1cf6-45d2-b374-54515798f714-kube-api-access-ctsmw\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:12.702685 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.702651    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:12.804442 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:12.804412    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:13.028840 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.028807    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:13.031194 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.031173    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls\") pod \"isvc-canary-create-v2-predictor-579784d7d7-wfkvd\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:13.129627 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.129595    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:13.131912 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.131891    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls\") pod \"isvc-canary-create-predictor-68984b9498-l98dp\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:13.143259 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.143230    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 20:00:13.232212 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.232180    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:13.308485 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.308458    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:13.506549 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.506525    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"]
Sep 18 20:00:13.508521 ip-10-0-141-190 kubenswrapper[2559]: W0918 20:00:13.508495    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261da5a3_9952_47a6_a373_3bdb58c7c8e1.slice/crio-9e367f93dbf6cb0aedcbc54c60dd8d4ebb9a3f5f4c0b87b23328471d35f6ef91 WatchSource:0}: Error finding container 9e367f93dbf6cb0aedcbc54c60dd8d4ebb9a3f5f4c0b87b23328471d35f6ef91: Status 404 returned error can't find the container with id 9e367f93dbf6cb0aedcbc54c60dd8d4ebb9a3f5f4c0b87b23328471d35f6ef91
Sep 18 20:00:13.581078 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:13.581051    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"]
Sep 18 20:00:13.582854 ip-10-0-141-190 kubenswrapper[2559]: W0918 20:00:13.582831    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16afa416_1cf6_45d2_b374_54515798f714.slice/crio-c3546316ce2a8632ffcecf5e0a806c9c0627b5b7fed6f39a2097025e5e043b4b WatchSource:0}: Error finding container c3546316ce2a8632ffcecf5e0a806c9c0627b5b7fed6f39a2097025e5e043b4b: Status 404 returned error can't find the container with id c3546316ce2a8632ffcecf5e0a806c9c0627b5b7fed6f39a2097025e5e043b4b
Sep 18 20:00:14.353909 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:14.353870    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" event={"ID":"261da5a3-9952-47a6-a373-3bdb58c7c8e1","Type":"ContainerStarted","Data":"fc19cddaca076bd9f6c96d1551f9df80b899cebdc99130bfdf9a07dc4210c557"}
Sep 18 20:00:14.354388 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:14.353942    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" event={"ID":"261da5a3-9952-47a6-a373-3bdb58c7c8e1","Type":"ContainerStarted","Data":"9e367f93dbf6cb0aedcbc54c60dd8d4ebb9a3f5f4c0b87b23328471d35f6ef91"}
Sep 18 20:00:14.355482 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:14.355454    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" event={"ID":"16afa416-1cf6-45d2-b374-54515798f714","Type":"ContainerStarted","Data":"b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f"}
Sep 18 20:00:14.355605 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:14.355513    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" event={"ID":"16afa416-1cf6-45d2-b374-54515798f714","Type":"ContainerStarted","Data":"c3546316ce2a8632ffcecf5e0a806c9c0627b5b7fed6f39a2097025e5e043b4b"}
Sep 18 20:00:15.747103 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:15.747058    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Sep 18 20:00:16.933199 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:16.933176    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 20:00:17.061008 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.060913    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba6a0aff-e782-488a-8949-474b9c442f33-kserve-provision-location\") pod \"ba6a0aff-e782-488a-8949-474b9c442f33\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") "
Sep 18 20:00:17.061008 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.060948    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-kube-rbac-proxy-sar-config\") pod \"ba6a0aff-e782-488a-8949-474b9c442f33\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") "
Sep 18 20:00:17.061232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.061014    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls\") pod \"ba6a0aff-e782-488a-8949-474b9c442f33\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") "
Sep 18 20:00:17.061232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.061031    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-cabundle-cert\") pod \"ba6a0aff-e782-488a-8949-474b9c442f33\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") "
Sep 18 20:00:17.061232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.061068    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x65vx\" (UniqueName: \"kubernetes.io/projected/ba6a0aff-e782-488a-8949-474b9c442f33-kube-api-access-x65vx\") pod \"ba6a0aff-e782-488a-8949-474b9c442f33\" (UID: \"ba6a0aff-e782-488a-8949-474b9c442f33\") "
Sep 18 20:00:17.061408 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.061259    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba6a0aff-e782-488a-8949-474b9c442f33-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba6a0aff-e782-488a-8949-474b9c442f33" (UID: "ba6a0aff-e782-488a-8949-474b9c442f33"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:17.061408 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.061379    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba6a0aff-e782-488a-8949-474b9c442f33" (UID: "ba6a0aff-e782-488a-8949-474b9c442f33"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:17.061504 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.061440    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba6a0aff-e782-488a-8949-474b9c442f33" (UID: "ba6a0aff-e782-488a-8949-474b9c442f33"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:17.063110 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.063081    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba6a0aff-e782-488a-8949-474b9c442f33" (UID: "ba6a0aff-e782-488a-8949-474b9c442f33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:00:17.063110 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.063084    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6a0aff-e782-488a-8949-474b9c442f33-kube-api-access-x65vx" (OuterVolumeSpecName: "kube-api-access-x65vx") pod "ba6a0aff-e782-488a-8949-474b9c442f33" (UID: "ba6a0aff-e782-488a-8949-474b9c442f33"). InnerVolumeSpecName "kube-api-access-x65vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:00:17.161790 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.161765    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba6a0aff-e782-488a-8949-474b9c442f33-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:17.161790 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.161788    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:17.161940 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.161797    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x65vx\" (UniqueName: \"kubernetes.io/projected/ba6a0aff-e782-488a-8949-474b9c442f33-kube-api-access-x65vx\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:17.161940 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.161807    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba6a0aff-e782-488a-8949-474b9c442f33-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:17.161940 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.161817    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba6a0aff-e782-488a-8949-474b9c442f33-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:17.369792 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.369693    2559 generic.go:356] "Generic (PLEG): container finished" podID="ba6a0aff-e782-488a-8949-474b9c442f33" containerID="972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee" exitCode=0
Sep 18 20:00:17.369792 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.369743    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" event={"ID":"ba6a0aff-e782-488a-8949-474b9c442f33","Type":"ContainerDied","Data":"972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee"}
Sep 18 20:00:17.369792 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.369762    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"
Sep 18 20:00:17.370107 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.369844    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p" event={"ID":"ba6a0aff-e782-488a-8949-474b9c442f33","Type":"ContainerDied","Data":"6777444c3c389c33980af11e1345beb36da1eb505e7b48f9951bb9465744cf79"}
Sep 18 20:00:17.370107 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.369888    2559 scope.go:117] "RemoveContainer" containerID="d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512"
Sep 18 20:00:17.378701 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.378676    2559 scope.go:117] "RemoveContainer" containerID="972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee"
Sep 18 20:00:17.387155 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.387132    2559 scope.go:117] "RemoveContainer" containerID="60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4"
Sep 18 20:00:17.389880 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.389857    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"]
Sep 18 20:00:17.394635 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.394611    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-1efdd-predictor-6f5d857b65-4sq8p"]
Sep 18 20:00:17.395503 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.395483    2559 scope.go:117] "RemoveContainer" containerID="d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512"
Sep 18 20:00:17.395813 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:17.395790    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512\": container with ID starting with d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512 not found: ID does not exist" containerID="d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512"
Sep 18 20:00:17.395896 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.395822    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512"} err="failed to get container status \"d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512\": rpc error: code = NotFound desc = could not find container \"d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512\": container with ID starting with d3f3b536bb39e83a48f79bb8198f98be3c1a6d3bcf081e2b640b7462193f1512 not found: ID does not exist"
Sep 18 20:00:17.395896 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.395839    2559 scope.go:117] "RemoveContainer" containerID="972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee"
Sep 18 20:00:17.396113 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:17.396095    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee\": container with ID starting with 972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee not found: ID does not exist" containerID="972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee"
Sep 18 20:00:17.396166 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.396119    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee"} err="failed to get container status \"972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee\": rpc error: code = NotFound desc = could not find container \"972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee\": container with ID starting with 972a9e6fd34e5a7d8f3921b5480d43bd6bd695f0c8a5177ed868415296ca16ee not found: ID does not exist"
Sep 18 20:00:17.396166 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.396133    2559 scope.go:117] "RemoveContainer" containerID="60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4"
Sep 18 20:00:17.396407 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:17.396387    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4\": container with ID starting with 60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4 not found: ID does not exist" containerID="60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4"
Sep 18 20:00:17.396463 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.396412    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4"} err="failed to get container status \"60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4\": rpc error: code = NotFound desc = could not find container \"60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4\": container with ID starting with 60559c7d843a76abe3091ebdbbe44054b30487ffcda9f4ae09abb50a50d8f6d4 not found: ID does not exist"
Sep 18 20:00:17.471245 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:17.471214    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" path="/var/lib/kubelet/pods/ba6a0aff-e782-488a-8949-474b9c442f33/volumes"
Sep 18 20:00:18.376366 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:18.376335    2559 generic.go:356] "Generic (PLEG): container finished" podID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerID="fc19cddaca076bd9f6c96d1551f9df80b899cebdc99130bfdf9a07dc4210c557" exitCode=0
Sep 18 20:00:18.376806 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:18.376416    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" event={"ID":"261da5a3-9952-47a6-a373-3bdb58c7c8e1","Type":"ContainerDied","Data":"fc19cddaca076bd9f6c96d1551f9df80b899cebdc99130bfdf9a07dc4210c557"}
Sep 18 20:00:18.378050 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:18.378022    2559 generic.go:356] "Generic (PLEG): container finished" podID="16afa416-1cf6-45d2-b374-54515798f714" containerID="b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f" exitCode=0
Sep 18 20:00:18.378154 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:18.378070    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" event={"ID":"16afa416-1cf6-45d2-b374-54515798f714","Type":"ContainerDied","Data":"b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f"}
Sep 18 20:00:19.383856 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.383823    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" event={"ID":"261da5a3-9952-47a6-a373-3bdb58c7c8e1","Type":"ContainerStarted","Data":"44644773c14c38de0144c9615961a3e2629f4ab5a6288c6d702f089c80a8e96d"}
Sep 18 20:00:19.384341 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.383884    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" event={"ID":"261da5a3-9952-47a6-a373-3bdb58c7c8e1","Type":"ContainerStarted","Data":"181e572924285899404ef7b16bfe4ac99d69e36e2673976099035ca967aed4b2"}
Sep 18 20:00:19.384341 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.384215    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:19.384453 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.384380    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:19.385765 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.385734    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:00:19.386055 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.386032    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" event={"ID":"16afa416-1cf6-45d2-b374-54515798f714","Type":"ContainerStarted","Data":"32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07"}
Sep 18 20:00:19.386129 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.386078    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" event={"ID":"16afa416-1cf6-45d2-b374-54515798f714","Type":"ContainerStarted","Data":"2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf"}
Sep 18 20:00:19.386258 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.386242    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:19.386317 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.386283    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:19.387337 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.387300    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:00:19.403232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.403191    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podStartSLOduration=3.586915943 podStartE2EDuration="7.403179112s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.816263169s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:19.401650426 +0000 UTC m=+1310.364383165" watchObservedRunningTime="2026-09-18 20:00:19.403179112 +0000 UTC m=+1310.365911850"
Sep 18 20:00:19.420187 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:19.420145    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podStartSLOduration=3.520958334 podStartE2EDuration="7.420131849s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899173515s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:00:19.41852474 +0000 UTC m=+1310.381257477" watchObservedRunningTime="2026-09-18 20:00:19.420131849 +0000 UTC m=+1310.382864586"
Sep 18 20:00:20.394871 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:20.394828    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:00:20.395261 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:20.394832    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:00:23.144250 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:23.144213    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 20:00:25.395847 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:25.395817    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:00:25.396607 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:25.396569    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:00:25.397061 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:25.397041    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:00:25.397463 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:25.397439    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:00:33.144749 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:33.144719    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 20:00:35.397181 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:35.397143    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:00:35.397638 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:35.397505    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:00:40.616510 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:40.616476    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"]
Sep 18 20:00:40.616915 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:40.616731    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" containerID="cri-o://12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf" gracePeriod=30
Sep 18 20:00:40.616915 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:40.616748    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kube-rbac-proxy" containerID="cri-o://c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a" gracePeriod=30
Sep 18 20:00:40.644946 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:40.644914    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:00:41.471875 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:41.471841    2559 generic.go:356] "Generic (PLEG): container finished" podID="19c05583-2a24-492b-8d80-f9022c3804ff" containerID="c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a" exitCode=2
Sep 18 20:00:41.471875 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:41.471878    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" event={"ID":"19c05583-2a24-492b-8d80-f9022c3804ff","Type":"ContainerDied","Data":"c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a"}
Sep 18 20:00:43.138533 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:43.138453    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Sep 18 20:00:43.143213 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:43.143190    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Sep 18 20:00:45.047972 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.047942    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 20:00:45.078214 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.078187    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-kube-rbac-proxy-sar-config\") pod \"19c05583-2a24-492b-8d80-f9022c3804ff\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") "
Sep 18 20:00:45.078326 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.078219    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c05583-2a24-492b-8d80-f9022c3804ff-proxy-tls\") pod \"19c05583-2a24-492b-8d80-f9022c3804ff\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") "
Sep 18 20:00:45.078326 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.078261    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-cabundle-cert\") pod \"19c05583-2a24-492b-8d80-f9022c3804ff\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") "
Sep 18 20:00:45.078326 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.078290    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-986l7\" (UniqueName: \"kubernetes.io/projected/19c05583-2a24-492b-8d80-f9022c3804ff-kube-api-access-986l7\") pod \"19c05583-2a24-492b-8d80-f9022c3804ff\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") "
Sep 18 20:00:45.078506 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.078343    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c05583-2a24-492b-8d80-f9022c3804ff-kserve-provision-location\") pod \"19c05583-2a24-492b-8d80-f9022c3804ff\" (UID: \"19c05583-2a24-492b-8d80-f9022c3804ff\") "
Sep 18 20:00:45.078647 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.078617    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19c05583-2a24-492b-8d80-f9022c3804ff" (UID: "19c05583-2a24-492b-8d80-f9022c3804ff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:45.078716 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.078638    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19c05583-2a24-492b-8d80-f9022c3804ff" (UID: "19c05583-2a24-492b-8d80-f9022c3804ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:00:45.078716 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.078695    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c05583-2a24-492b-8d80-f9022c3804ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19c05583-2a24-492b-8d80-f9022c3804ff" (UID: "19c05583-2a24-492b-8d80-f9022c3804ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:00:45.080307 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.080282    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c05583-2a24-492b-8d80-f9022c3804ff-kube-api-access-986l7" (OuterVolumeSpecName: "kube-api-access-986l7") pod "19c05583-2a24-492b-8d80-f9022c3804ff" (UID: "19c05583-2a24-492b-8d80-f9022c3804ff"). InnerVolumeSpecName "kube-api-access-986l7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:00:45.080459 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.080442    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19c05583-2a24-492b-8d80-f9022c3804ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19c05583-2a24-492b-8d80-f9022c3804ff" (UID: "19c05583-2a24-492b-8d80-f9022c3804ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:00:45.178768 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.178718    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19c05583-2a24-492b-8d80-f9022c3804ff-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:45.178768 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.178736    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:45.178768 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.178746    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19c05583-2a24-492b-8d80-f9022c3804ff-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:45.178768 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.178754    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19c05583-2a24-492b-8d80-f9022c3804ff-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:45.178768 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.178762    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-986l7\" (UniqueName: \"kubernetes.io/projected/19c05583-2a24-492b-8d80-f9022c3804ff-kube-api-access-986l7\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:00:45.396474 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.396443    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:00:45.397655 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.397632    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:00:45.487687 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.487663    2559 generic.go:356] "Generic (PLEG): container finished" podID="19c05583-2a24-492b-8d80-f9022c3804ff" containerID="12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf" exitCode=0
Sep 18 20:00:45.487789 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.487700    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" event={"ID":"19c05583-2a24-492b-8d80-f9022c3804ff","Type":"ContainerDied","Data":"12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf"}
Sep 18 20:00:45.487789 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.487710    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"
Sep 18 20:00:45.487789 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.487740    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m" event={"ID":"19c05583-2a24-492b-8d80-f9022c3804ff","Type":"ContainerDied","Data":"d5ef9eb25785c2d1d9d821bd5c249a9ca96ebc1b0ab271319b516646fe813d62"}
Sep 18 20:00:45.487789 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.487764    2559 scope.go:117] "RemoveContainer" containerID="c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a"
Sep 18 20:00:45.496057 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.496037    2559 scope.go:117] "RemoveContainer" containerID="12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf"
Sep 18 20:00:45.502899 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.502883    2559 scope.go:117] "RemoveContainer" containerID="a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea"
Sep 18 20:00:45.507032 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.507015    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"]
Sep 18 20:00:45.510256 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.510149    2559 scope.go:117] "RemoveContainer" containerID="c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a"
Sep 18 20:00:45.510501 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:45.510479    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a\": container with ID starting with c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a not found: ID does not exist" containerID="c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a"
Sep 18 20:00:45.510570 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.510510    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a"} err="failed to get container status \"c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a\": rpc error: code = NotFound desc = could not find container \"c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a\": container with ID starting with c4044536bf52b34016a2d33a55e64143a38f8f428b027f0c0b715dab0ee8119a not found: ID does not exist"
Sep 18 20:00:45.510570 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.510527    2559 scope.go:117] "RemoveContainer" containerID="12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf"
Sep 18 20:00:45.510831 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:45.510808    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf\": container with ID starting with 12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf not found: ID does not exist" containerID="12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf"
Sep 18 20:00:45.510890 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.510841    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf"} err="failed to get container status \"12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf\": rpc error: code = NotFound desc = could not find container \"12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf\": container with ID starting with 12f124e1461d4c6313e598bf79b7d92d27dcceb5c9fa32bbe7af0c02a9a87baf not found: ID does not exist"
Sep 18 20:00:45.510890 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.510862    2559 scope.go:117] "RemoveContainer" containerID="a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea"
Sep 18 20:00:45.511155 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:00:45.511128    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea\": container with ID starting with a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea not found: ID does not exist" containerID="a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea"
Sep 18 20:00:45.511259 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.511155    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea"} err="failed to get container status \"a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea\": rpc error: code = NotFound desc = could not find container \"a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea\": container with ID starting with a0bc25dc81c024937f499d4049ae646017c6607e0fc73cf63bcecdc1c0bdb8ea not found: ID does not exist"
Sep 18 20:00:45.511455 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:45.511440    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-aa327-predictor-7dfdcb6cb7-6ls7m"]
Sep 18 20:00:47.469535 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:47.469505    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" path="/var/lib/kubelet/pods/19c05583-2a24-492b-8d80-f9022c3804ff/volumes"
Sep 18 20:00:55.396792 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:55.396729    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:00:55.398160 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:00:55.398136    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:01:05.397296 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:05.397255    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:01:05.397687 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:05.397413    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:01:15.396637 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:15.396592    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:01:15.397903 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:15.397869    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:01:24.659481 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:24.659450    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:25.396492 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:25.396458    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:01:25.397670 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:25.397647    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:01:35.397124 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:35.397093    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:01:35.398063 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:35.398042    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:01:42.383066 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.383033    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"]
Sep 18 20:01:42.383467 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.383277    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" containerID="cri-o://181e572924285899404ef7b16bfe4ac99d69e36e2673976099035ca967aed4b2" gracePeriod=30
Sep 18 20:01:42.383467 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.383326    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kube-rbac-proxy" containerID="cri-o://44644773c14c38de0144c9615961a3e2629f4ab5a6288c6d702f089c80a8e96d" gracePeriod=30
Sep 18 20:01:42.388320 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.388293    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"]
Sep 18 20:01:42.388698 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.388538    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" containerID="cri-o://2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf" gracePeriod=30
Sep 18 20:01:42.388698 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.388599    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kube-rbac-proxy" containerID="cri-o://32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07" gracePeriod=30
Sep 18 20:01:42.403886 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.403865    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:42.459870 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.459847    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:42.696404 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.696330    2559 generic.go:356] "Generic (PLEG): container finished" podID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerID="44644773c14c38de0144c9615961a3e2629f4ab5a6288c6d702f089c80a8e96d" exitCode=2
Sep 18 20:01:42.696546 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.696399    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" event={"ID":"261da5a3-9952-47a6-a373-3bdb58c7c8e1","Type":"ContainerDied","Data":"44644773c14c38de0144c9615961a3e2629f4ab5a6288c6d702f089c80a8e96d"}
Sep 18 20:01:42.698472 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.698449    2559 generic.go:356] "Generic (PLEG): container finished" podID="16afa416-1cf6-45d2-b374-54515798f714" containerID="32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07" exitCode=2
Sep 18 20:01:42.698596 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:42.698510    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" event={"ID":"16afa416-1cf6-45d2-b374-54515798f714","Type":"ContainerDied","Data":"32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07"}
Sep 18 20:01:45.391327 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:45.391284    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 18 20:01:45.393245 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:45.393221    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 18 20:01:45.396603 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:45.396579    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Sep 18 20:01:45.397859 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:45.397835    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 18 20:01:46.636481 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.636459    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:01:46.696755 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.696732    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls\") pod \"16afa416-1cf6-45d2-b374-54515798f714\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") "
Sep 18 20:01:46.696889 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.696792    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-cabundle-cert\") pod \"16afa416-1cf6-45d2-b374-54515798f714\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") "
Sep 18 20:01:46.696889 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.696819    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctsmw\" (UniqueName: \"kubernetes.io/projected/16afa416-1cf6-45d2-b374-54515798f714-kube-api-access-ctsmw\") pod \"16afa416-1cf6-45d2-b374-54515798f714\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") "
Sep 18 20:01:46.696889 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.696855    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16afa416-1cf6-45d2-b374-54515798f714-kserve-provision-location\") pod \"16afa416-1cf6-45d2-b374-54515798f714\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") "
Sep 18 20:01:46.697130 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.696893    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-kube-rbac-proxy-sar-config\") pod \"16afa416-1cf6-45d2-b374-54515798f714\" (UID: \"16afa416-1cf6-45d2-b374-54515798f714\") "
Sep 18 20:01:46.697284 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.697196    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16afa416-1cf6-45d2-b374-54515798f714-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16afa416-1cf6-45d2-b374-54515798f714" (UID: "16afa416-1cf6-45d2-b374-54515798f714"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:46.697284 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.697277    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "16afa416-1cf6-45d2-b374-54515798f714" (UID: "16afa416-1cf6-45d2-b374-54515798f714"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:46.697454 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.697314    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "16afa416-1cf6-45d2-b374-54515798f714" (UID: "16afa416-1cf6-45d2-b374-54515798f714"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:46.699843 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.699816    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16afa416-1cf6-45d2-b374-54515798f714-kube-api-access-ctsmw" (OuterVolumeSpecName: "kube-api-access-ctsmw") pod "16afa416-1cf6-45d2-b374-54515798f714" (UID: "16afa416-1cf6-45d2-b374-54515798f714"). InnerVolumeSpecName "kube-api-access-ctsmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:46.699931 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.699886    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "16afa416-1cf6-45d2-b374-54515798f714" (UID: "16afa416-1cf6-45d2-b374-54515798f714"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:46.714697 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.714676    2559 generic.go:356] "Generic (PLEG): container finished" podID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerID="181e572924285899404ef7b16bfe4ac99d69e36e2673976099035ca967aed4b2" exitCode=0
Sep 18 20:01:46.714796 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.714752    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" event={"ID":"261da5a3-9952-47a6-a373-3bdb58c7c8e1","Type":"ContainerDied","Data":"181e572924285899404ef7b16bfe4ac99d69e36e2673976099035ca967aed4b2"}
Sep 18 20:01:46.716920 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.716899    2559 generic.go:356] "Generic (PLEG): container finished" podID="16afa416-1cf6-45d2-b374-54515798f714" containerID="2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf" exitCode=0
Sep 18 20:01:46.717015 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.716933    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" event={"ID":"16afa416-1cf6-45d2-b374-54515798f714","Type":"ContainerDied","Data":"2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf"}
Sep 18 20:01:46.717015 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.716975    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"
Sep 18 20:01:46.717015 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.717002    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp" event={"ID":"16afa416-1cf6-45d2-b374-54515798f714","Type":"ContainerDied","Data":"c3546316ce2a8632ffcecf5e0a806c9c0627b5b7fed6f39a2097025e5e043b4b"}
Sep 18 20:01:46.717125 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.717042    2559 scope.go:117] "RemoveContainer" containerID="32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07"
Sep 18 20:01:46.726946 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.726930    2559 scope.go:117] "RemoveContainer" containerID="2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf"
Sep 18 20:01:46.727594 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.727576    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:01:46.734184 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.734167    2559 scope.go:117] "RemoveContainer" containerID="b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f"
Sep 18 20:01:46.740276 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.740256    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"]
Sep 18 20:01:46.741045 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.741033    2559 scope.go:117] "RemoveContainer" containerID="32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07"
Sep 18 20:01:46.741316 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:01:46.741299    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07\": container with ID starting with 32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07 not found: ID does not exist" containerID="32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07"
Sep 18 20:01:46.741379 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.741320    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07"} err="failed to get container status \"32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07\": rpc error: code = NotFound desc = could not find container \"32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07\": container with ID starting with 32af81b66dd2dbf69c601d6448b78d03629997518119290f449c24239c17ac07 not found: ID does not exist"
Sep 18 20:01:46.741379 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.741335    2559 scope.go:117] "RemoveContainer" containerID="2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf"
Sep 18 20:01:46.741545 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:01:46.741530    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf\": container with ID starting with 2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf not found: ID does not exist" containerID="2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf"
Sep 18 20:01:46.741585 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.741549    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf"} err="failed to get container status \"2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf\": rpc error: code = NotFound desc = could not find container \"2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf\": container with ID starting with 2d896a5f0c5726dc985a59b3de6fabd3d3d9c7b60da2a7df902d3202e59c4ecf not found: ID does not exist"
Sep 18 20:01:46.741585 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.741562    2559 scope.go:117] "RemoveContainer" containerID="b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f"
Sep 18 20:01:46.741786 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:01:46.741771    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f\": container with ID starting with b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f not found: ID does not exist" containerID="b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f"
Sep 18 20:01:46.741835 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.741790    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f"} err="failed to get container status \"b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f\": rpc error: code = NotFound desc = could not find container \"b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f\": container with ID starting with b2f76bcbade37d82f1500538a3e88955d70700c53a7649ec196cb33d534dc12f not found: ID does not exist"
Sep 18 20:01:46.744613 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.744594    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-68984b9498-l98dp"]
Sep 18 20:01:46.798199 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798177    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-rbac-proxy-sar-config\") pod \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") "
Sep 18 20:01:46.798272 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798235    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kserve-provision-location\") pod \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") "
Sep 18 20:01:46.798272 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798254    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls\") pod \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") "
Sep 18 20:01:46.798367 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798271    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnsbz\" (UniqueName: \"kubernetes.io/projected/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-api-access-vnsbz\") pod \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") "
Sep 18 20:01:46.798367 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798291    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-cabundle-cert\") pod \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\" (UID: \"261da5a3-9952-47a6-a373-3bdb58c7c8e1\") "
Sep 18 20:01:46.798473 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798416    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/16afa416-1cf6-45d2-b374-54515798f714-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.798473 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798433    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.798473 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798445    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctsmw\" (UniqueName: \"kubernetes.io/projected/16afa416-1cf6-45d2-b374-54515798f714-kube-api-access-ctsmw\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.798473 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798457    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16afa416-1cf6-45d2-b374-54515798f714-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.798473 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798471    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/16afa416-1cf6-45d2-b374-54515798f714-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.798703 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798566    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "261da5a3-9952-47a6-a373-3bdb58c7c8e1" (UID: "261da5a3-9952-47a6-a373-3bdb58c7c8e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:46.798703 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798620    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "261da5a3-9952-47a6-a373-3bdb58c7c8e1" (UID: "261da5a3-9952-47a6-a373-3bdb58c7c8e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:01:46.798703 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.798661    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "261da5a3-9952-47a6-a373-3bdb58c7c8e1" (UID: "261da5a3-9952-47a6-a373-3bdb58c7c8e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:01:46.800244 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.800227    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "261da5a3-9952-47a6-a373-3bdb58c7c8e1" (UID: "261da5a3-9952-47a6-a373-3bdb58c7c8e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:01:46.800306 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.800247    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-api-access-vnsbz" (OuterVolumeSpecName: "kube-api-access-vnsbz") pod "261da5a3-9952-47a6-a373-3bdb58c7c8e1" (UID: "261da5a3-9952-47a6-a373-3bdb58c7c8e1"). InnerVolumeSpecName "kube-api-access-vnsbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:01:46.899693 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.899669    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnsbz\" (UniqueName: \"kubernetes.io/projected/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-api-access-vnsbz\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.899693 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.899691    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.899815 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.899701    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.899815 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.899710    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/261da5a3-9952-47a6-a373-3bdb58c7c8e1-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:46.899815 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:46.899719    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/261da5a3-9952-47a6-a373-3bdb58c7c8e1-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:01:47.470710 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.470683    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16afa416-1cf6-45d2-b374-54515798f714" path="/var/lib/kubelet/pods/16afa416-1cf6-45d2-b374-54515798f714/volumes"
Sep 18 20:01:47.508461 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.508436    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:47.566877 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.566853    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"]
Sep 18 20:01:47.567109 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567097    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567153 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567111    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567153 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567122    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567153 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567127    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567153 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567133    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567153 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567139    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567153 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567147    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="storage-initializer"
Sep 18 20:01:47.567153 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567152    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="storage-initializer"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567160    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567165    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567172    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567176    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567183    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="storage-initializer"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567188    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="storage-initializer"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567199    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="storage-initializer"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567203    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="storage-initializer"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567209    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567215    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567229    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567233    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567240    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="storage-initializer"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567245    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="storage-initializer"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567252    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567257    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567296    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567305    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567311    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567316    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c05583-2a24-492b-8d80-f9022c3804ff" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567321    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="16afa416-1cf6-45d2-b374-54515798f714" containerName="kserve-container"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567325    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567330    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" containerName="kube-rbac-proxy"
Sep 18 20:01:47.567368 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.567335    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6a0aff-e782-488a-8949-474b9c442f33" containerName="kserve-container"
Sep 18 20:01:47.573723 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.573699    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.575945 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.575903    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 18 20:01:47.576220 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.576203    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 18 20:01:47.577184 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.577163    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"]
Sep 18 20:01:47.580160 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.580139    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"]
Sep 18 20:01:47.580283 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.580265    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.582174 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.582154    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 18 20:01:47.583905 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.583883    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:47.591752 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.591734    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"]
Sep 18 20:01:47.704191 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704161    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704198    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53863-2ece-4079-8590-ba1b442137c2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704227    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53863-2ece-4079-8590-ba1b442137c2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704261    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9992\" (UniqueName: \"kubernetes.io/projected/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-api-access-z9992\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704302    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704364    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8j4\" (UniqueName: \"kubernetes.io/projected/7fa53863-2ece-4079-8590-ba1b442137c2-kube-api-access-kb8j4\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704391    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704439    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704466    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.704558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.704502    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.722068 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.722000    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd" event={"ID":"261da5a3-9952-47a6-a373-3bdb58c7c8e1","Type":"ContainerDied","Data":"9e367f93dbf6cb0aedcbc54c60dd8d4ebb9a3f5f4c0b87b23328471d35f6ef91"}
Sep 18 20:01:47.722068 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.722019    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"
Sep 18 20:01:47.722223 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.722068    2559 scope.go:117] "RemoveContainer" containerID="44644773c14c38de0144c9615961a3e2629f4ab5a6288c6d702f089c80a8e96d"
Sep 18 20:01:47.729434 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.729416    2559 scope.go:117] "RemoveContainer" containerID="181e572924285899404ef7b16bfe4ac99d69e36e2673976099035ca967aed4b2"
Sep 18 20:01:47.736313 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.736294    2559 scope.go:117] "RemoveContainer" containerID="fc19cddaca076bd9f6c96d1551f9df80b899cebdc99130bfdf9a07dc4210c557"
Sep 18 20:01:47.739280 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.739253    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"]
Sep 18 20:01:47.744212 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.744195    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-579784d7d7-wfkvd"]
Sep 18 20:01:47.785590 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.785565    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:47.805345 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805323    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.805445 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805350    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.805445 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805368    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.805445 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805390    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.805445 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805406    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53863-2ece-4079-8590-ba1b442137c2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.805673 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805609    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53863-2ece-4079-8590-ba1b442137c2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.805673 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805654    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9992\" (UniqueName: \"kubernetes.io/projected/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-api-access-z9992\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.805780 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805699    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.805780 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805748    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8j4\" (UniqueName: \"kubernetes.io/projected/7fa53863-2ece-4079-8590-ba1b442137c2-kube-api-access-kb8j4\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.805780 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805766    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.805935 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805775    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.805935 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.805917    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53863-2ece-4079-8590-ba1b442137c2-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.806178 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.806154    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.806253 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.806203    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.806435 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.806405    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-cabundle-cert\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.806435 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.806410    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.807771 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.807753    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53863-2ece-4079-8590-ba1b442137c2-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.808114 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.808099    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-proxy-tls\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.813256 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.813235    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9992\" (UniqueName: \"kubernetes.io/projected/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-api-access-z9992\") pod \"isvc-canary-promote-predictor-cb75746f8-grg87\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:47.813765 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.813741    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8j4\" (UniqueName: \"kubernetes.io/projected/7fa53863-2ece-4079-8590-ba1b442137c2-kube-api-access-kb8j4\") pod \"isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.885211 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.885189    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:47.886475 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.886456    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:01:47.892474 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:47.892462    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:48.155224 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:48.155202    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"]
Sep 18 20:01:48.157328 ip-10-0-141-190 kubenswrapper[2559]: W0918 20:01:48.157303    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa53863_2ece_4079_8590_ba1b442137c2.slice/crio-bb3431fd9ec3d4fa78a0ccb26cff138b1615aa5fcb15414d7a1b97ac4543174c WatchSource:0}: Error finding container bb3431fd9ec3d4fa78a0ccb26cff138b1615aa5fcb15414d7a1b97ac4543174c: Status 404 returned error can't find the container with id bb3431fd9ec3d4fa78a0ccb26cff138b1615aa5fcb15414d7a1b97ac4543174c
Sep 18 20:01:48.162278 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:48.162255    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"]
Sep 18 20:01:48.164359 ip-10-0-141-190 kubenswrapper[2559]: W0918 20:01:48.164261    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c248f96_4bea_454e_b5c9_8c144cfa7ee4.slice/crio-18472842574ae3f6511aa06f79b203be66c6098521efe1caa7abdcdf2ee42185 WatchSource:0}: Error finding container 18472842574ae3f6511aa06f79b203be66c6098521efe1caa7abdcdf2ee42185: Status 404 returned error can't find the container with id 18472842574ae3f6511aa06f79b203be66c6098521efe1caa7abdcdf2ee42185
Sep 18 20:01:48.729122 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:48.729079    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" event={"ID":"0c248f96-4bea-454e-b5c9-8c144cfa7ee4","Type":"ContainerStarted","Data":"fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e"}
Sep 18 20:01:48.729575 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:48.729165    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" event={"ID":"0c248f96-4bea-454e-b5c9-8c144cfa7ee4","Type":"ContainerStarted","Data":"18472842574ae3f6511aa06f79b203be66c6098521efe1caa7abdcdf2ee42185"}
Sep 18 20:01:48.730852 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:48.730829    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" event={"ID":"7fa53863-2ece-4079-8590-ba1b442137c2","Type":"ContainerStarted","Data":"6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01"}
Sep 18 20:01:48.731012 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:48.730888    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" event={"ID":"7fa53863-2ece-4079-8590-ba1b442137c2","Type":"ContainerStarted","Data":"bb3431fd9ec3d4fa78a0ccb26cff138b1615aa5fcb15414d7a1b97ac4543174c"}
Sep 18 20:01:49.469619 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:49.469575    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="261da5a3-9952-47a6-a373-3bdb58c7c8e1" path="/var/lib/kubelet/pods/261da5a3-9952-47a6-a373-3bdb58c7c8e1/volumes"
Sep 18 20:01:52.748642 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:52.748598    2559 generic.go:356] "Generic (PLEG): container finished" podID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerID="fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e" exitCode=0
Sep 18 20:01:52.749123 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:52.748675    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" event={"ID":"0c248f96-4bea-454e-b5c9-8c144cfa7ee4","Type":"ContainerDied","Data":"fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e"}
Sep 18 20:01:52.750836 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:52.750810    2559 generic.go:356] "Generic (PLEG): container finished" podID="7fa53863-2ece-4079-8590-ba1b442137c2" containerID="6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01" exitCode=0
Sep 18 20:01:52.751002 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:52.750885    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" event={"ID":"7fa53863-2ece-4079-8590-ba1b442137c2","Type":"ContainerDied","Data":"6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01"}
Sep 18 20:01:53.757642 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.757607    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" event={"ID":"7fa53863-2ece-4079-8590-ba1b442137c2","Type":"ContainerStarted","Data":"ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c"}
Sep 18 20:01:53.758092 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.757685    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" event={"ID":"7fa53863-2ece-4079-8590-ba1b442137c2","Type":"ContainerStarted","Data":"841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3"}
Sep 18 20:01:53.758092 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.758050    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:53.758228 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.758105    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:53.759622 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.759570    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:01:53.759973 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.759940    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" event={"ID":"0c248f96-4bea-454e-b5c9-8c144cfa7ee4","Type":"ContainerStarted","Data":"8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601"}
Sep 18 20:01:53.760045 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.760008    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" event={"ID":"0c248f96-4bea-454e-b5c9-8c144cfa7ee4","Type":"ContainerStarted","Data":"fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89"}
Sep 18 20:01:53.760263 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.760243    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:53.777418 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.777376    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podStartSLOduration=3.155066239 podStartE2EDuration="6.777364833s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.622298594s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:53.77586038 +0000 UTC m=+1404.738593143" watchObservedRunningTime="2026-09-18 20:01:53.777364833 +0000 UTC m=+1404.740097571"
Sep 18 20:01:53.792759 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:53.792715    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podStartSLOduration=3.264511469 podStartE2EDuration="6.792702704s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.528191235s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:01:53.791638228 +0000 UTC m=+1404.754370968" watchObservedRunningTime="2026-09-18 20:01:53.792702704 +0000 UTC m=+1404.755435442"
Sep 18 20:01:54.763857 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:54.763817    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:01:54.764281 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:54.763998    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:01:54.765333 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:54.765310    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:01:55.767884 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:55.767843    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:01:59.768377 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:59.768346    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:01:59.768862 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:01:59.768838    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:02:00.772265 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:00.772241    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:02:00.772768 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:00.772741    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:02:09.769557 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:09.769514    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:02:10.773207 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:10.773160    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:02:19.768857 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:19.768766    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:02:20.772786 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:20.772746    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:02:29.769319 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:29.769275    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:02:30.773233 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:30.773194    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:02:39.769272 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:39.769235    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:02:40.773271 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:40.773231    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:02:49.768805 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:49.768760    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:02:50.772906 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:50.772866    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:02:59.769714 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:02:59.769673    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:03:00.773113 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:00.773076    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 18 20:03:01.467336 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:01.467291    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:03:10.465906 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:10.465871    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:03:11.470870 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:11.470842    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:03:11.583619 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:11.583581    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"]
Sep 18 20:03:12.053133 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:12.053097    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kube-rbac-proxy" containerID="cri-o://8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601" gracePeriod=30
Sep 18 20:03:12.053370 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:12.053094    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container" containerID="cri-o://fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89" gracePeriod=30
Sep 18 20:03:13.059323 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:13.059289    2559 generic.go:356] "Generic (PLEG): container finished" podID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerID="8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601" exitCode=2
Sep 18 20:03:13.059703 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:13.059361    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" event={"ID":"0c248f96-4bea-454e-b5c9-8c144cfa7ee4","Type":"ContainerDied","Data":"8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601"}
Sep 18 20:03:15.768847 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:15.768795    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 18 20:03:16.794088 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.794056    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:03:16.875465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.875425    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-cabundle-cert\") pod \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") "
Sep 18 20:03:16.875658 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.875485    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-proxy-tls\") pod \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") "
Sep 18 20:03:16.875658 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.875506    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9992\" (UniqueName: \"kubernetes.io/projected/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-api-access-z9992\") pod \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") "
Sep 18 20:03:16.875658 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.875547    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kserve-provision-location\") pod \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") "
Sep 18 20:03:16.875658 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.875605    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-rbac-proxy-sar-config\") pod \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\" (UID: \"0c248f96-4bea-454e-b5c9-8c144cfa7ee4\") "
Sep 18 20:03:16.875935 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.875907    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c248f96-4bea-454e-b5c9-8c144cfa7ee4" (UID: "0c248f96-4bea-454e-b5c9-8c144cfa7ee4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:16.875935 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.875918    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c248f96-4bea-454e-b5c9-8c144cfa7ee4" (UID: "0c248f96-4bea-454e-b5c9-8c144cfa7ee4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:03:16.876054 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.876036    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c248f96-4bea-454e-b5c9-8c144cfa7ee4" (UID: "0c248f96-4bea-454e-b5c9-8c144cfa7ee4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:03:16.877662 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.877635    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-api-access-z9992" (OuterVolumeSpecName: "kube-api-access-z9992") pod "0c248f96-4bea-454e-b5c9-8c144cfa7ee4" (UID: "0c248f96-4bea-454e-b5c9-8c144cfa7ee4"). InnerVolumeSpecName "kube-api-access-z9992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:03:16.877662 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.877643    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c248f96-4bea-454e-b5c9-8c144cfa7ee4" (UID: "0c248f96-4bea-454e-b5c9-8c144cfa7ee4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:03:16.976777 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.976741    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:03:16.976777 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.976774    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:03:16.976777 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.976784    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:03:16.977032 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.976793    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9992\" (UniqueName: \"kubernetes.io/projected/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kube-api-access-z9992\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:03:16.977032 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:16.976801    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c248f96-4bea-454e-b5c9-8c144cfa7ee4-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:03:17.076311 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.076277    2559 generic.go:356] "Generic (PLEG): container finished" podID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerID="fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89" exitCode=0
Sep 18 20:03:17.076311 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.076318    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" event={"ID":"0c248f96-4bea-454e-b5c9-8c144cfa7ee4","Type":"ContainerDied","Data":"fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89"}
Sep 18 20:03:17.076559 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.076351    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"
Sep 18 20:03:17.076559 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.076370    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87" event={"ID":"0c248f96-4bea-454e-b5c9-8c144cfa7ee4","Type":"ContainerDied","Data":"18472842574ae3f6511aa06f79b203be66c6098521efe1caa7abdcdf2ee42185"}
Sep 18 20:03:17.076559 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.076399    2559 scope.go:117] "RemoveContainer" containerID="8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601"
Sep 18 20:03:17.085356 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.085338    2559 scope.go:117] "RemoveContainer" containerID="fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89"
Sep 18 20:03:17.092433 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.092418    2559 scope.go:117] "RemoveContainer" containerID="fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e"
Sep 18 20:03:17.097890 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.097868    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"]
Sep 18 20:03:17.099610 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.099595    2559 scope.go:117] "RemoveContainer" containerID="8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601"
Sep 18 20:03:17.099887 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:03:17.099869    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601\": container with ID starting with 8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601 not found: ID does not exist" containerID="8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601"
Sep 18 20:03:17.099937 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.099898    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601"} err="failed to get container status \"8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601\": rpc error: code = NotFound desc = could not find container \"8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601\": container with ID starting with 8cb77429ffbaeda72cce63f41a295c551239900df3f020d5eb9ea14a8c374601 not found: ID does not exist"
Sep 18 20:03:17.099937 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.099916    2559 scope.go:117] "RemoveContainer" containerID="fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89"
Sep 18 20:03:17.100173 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:03:17.100156    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89\": container with ID starting with fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89 not found: ID does not exist" containerID="fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89"
Sep 18 20:03:17.100219 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.100181    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89"} err="failed to get container status \"fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89\": rpc error: code = NotFound desc = could not find container \"fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89\": container with ID starting with fa6fe4c92b02282881a9dbc4f92a7494893047bd142fb9f99a560ba42b527f89 not found: ID does not exist"
Sep 18 20:03:17.100219 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.100197    2559 scope.go:117] "RemoveContainer" containerID="fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e"
Sep 18 20:03:17.100410 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:03:17.100396    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e\": container with ID starting with fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e not found: ID does not exist" containerID="fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e"
Sep 18 20:03:17.100457 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.100413    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e"} err="failed to get container status \"fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e\": rpc error: code = NotFound desc = could not find container \"fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e\": container with ID starting with fc7b8dd26a7b785ca1a48f5ff4bd37639b3e0d73de9bca4f8c42bf2bc838965e not found: ID does not exist"
Sep 18 20:03:17.104271 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.104251    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-cb75746f8-grg87"]
Sep 18 20:03:17.469706 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:17.469674    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" path="/var/lib/kubelet/pods/0c248f96-4bea-454e-b5c9-8c144cfa7ee4/volumes"
Sep 18 20:03:18.423258 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:18.423227    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:03:31.437653 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:31.437621    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 20:03:31.438189 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:31.437630    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 20:03:41.398977 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:03:41.398940    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:47.958769 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:47.958738    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"]
Sep 18 20:04:47.959328 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:47.958987    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" containerID="cri-o://841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3" gracePeriod=30
Sep 18 20:04:47.959328 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:47.959037    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kube-rbac-proxy" containerID="cri-o://ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c" gracePeriod=30
Sep 18 20:04:47.981353 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:47.981316    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:04:48.412682 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:48.412596    2559 generic.go:356] "Generic (PLEG): container finished" podID="7fa53863-2ece-4079-8590-ba1b442137c2" containerID="ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c" exitCode=2
Sep 18 20:04:48.412682 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:48.412633    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" event={"ID":"7fa53863-2ece-4079-8590-ba1b442137c2","Type":"ContainerDied","Data":"ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c"}
Sep 18 20:04:49.764301 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:49.764252    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 18 20:04:51.467508 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.467480    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 18 20:04:51.594295 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.594274    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:04:51.670520 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.670497    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53863-2ece-4079-8590-ba1b442137c2-kserve-provision-location\") pod \"7fa53863-2ece-4079-8590-ba1b442137c2\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") "
Sep 18 20:04:51.670623 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.670566    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53863-2ece-4079-8590-ba1b442137c2-proxy-tls\") pod \"7fa53863-2ece-4079-8590-ba1b442137c2\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") "
Sep 18 20:04:51.670623 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.670603    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-kube-rbac-proxy-sar-config\") pod \"7fa53863-2ece-4079-8590-ba1b442137c2\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") "
Sep 18 20:04:51.670707 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.670630    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-cabundle-cert\") pod \"7fa53863-2ece-4079-8590-ba1b442137c2\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") "
Sep 18 20:04:51.670707 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.670676    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb8j4\" (UniqueName: \"kubernetes.io/projected/7fa53863-2ece-4079-8590-ba1b442137c2-kube-api-access-kb8j4\") pod \"7fa53863-2ece-4079-8590-ba1b442137c2\" (UID: \"7fa53863-2ece-4079-8590-ba1b442137c2\") "
Sep 18 20:04:51.670835 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.670815    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa53863-2ece-4079-8590-ba1b442137c2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fa53863-2ece-4079-8590-ba1b442137c2" (UID: "7fa53863-2ece-4079-8590-ba1b442137c2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:04:51.670984 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.670942    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7fa53863-2ece-4079-8590-ba1b442137c2" (UID: "7fa53863-2ece-4079-8590-ba1b442137c2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:51.671073 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.670978    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa53863-2ece-4079-8590-ba1b442137c2-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:04:51.671073 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.671037    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fa53863-2ece-4079-8590-ba1b442137c2" (UID: "7fa53863-2ece-4079-8590-ba1b442137c2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:04:51.672581 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.672556    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa53863-2ece-4079-8590-ba1b442137c2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fa53863-2ece-4079-8590-ba1b442137c2" (UID: "7fa53863-2ece-4079-8590-ba1b442137c2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:04:51.672681 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.672652    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa53863-2ece-4079-8590-ba1b442137c2-kube-api-access-kb8j4" (OuterVolumeSpecName: "kube-api-access-kb8j4") pod "7fa53863-2ece-4079-8590-ba1b442137c2" (UID: "7fa53863-2ece-4079-8590-ba1b442137c2"). InnerVolumeSpecName "kube-api-access-kb8j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:04:51.771283 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.771221    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa53863-2ece-4079-8590-ba1b442137c2-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:04:51.771283 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.771242    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:04:51.771283 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.771257    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa53863-2ece-4079-8590-ba1b442137c2-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:04:51.771283 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:51.771269    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb8j4\" (UniqueName: \"kubernetes.io/projected/7fa53863-2ece-4079-8590-ba1b442137c2-kube-api-access-kb8j4\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:04:52.429794 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.429758    2559 generic.go:356] "Generic (PLEG): container finished" podID="7fa53863-2ece-4079-8590-ba1b442137c2" containerID="841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3" exitCode=0
Sep 18 20:04:52.430003 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.429836    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"
Sep 18 20:04:52.430003 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.429850    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" event={"ID":"7fa53863-2ece-4079-8590-ba1b442137c2","Type":"ContainerDied","Data":"841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3"}
Sep 18 20:04:52.430003 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.429920    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p" event={"ID":"7fa53863-2ece-4079-8590-ba1b442137c2","Type":"ContainerDied","Data":"bb3431fd9ec3d4fa78a0ccb26cff138b1615aa5fcb15414d7a1b97ac4543174c"}
Sep 18 20:04:52.430003 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.429949    2559 scope.go:117] "RemoveContainer" containerID="ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c"
Sep 18 20:04:52.439240 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.439224    2559 scope.go:117] "RemoveContainer" containerID="841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3"
Sep 18 20:04:52.446674 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.446657    2559 scope.go:117] "RemoveContainer" containerID="6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01"
Sep 18 20:04:52.451060 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.451041    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"]
Sep 18 20:04:52.454446 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.454396    2559 scope.go:117] "RemoveContainer" containerID="ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c"
Sep 18 20:04:52.454746 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:04:52.454722    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c\": container with ID starting with ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c not found: ID does not exist" containerID="ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c"
Sep 18 20:04:52.454855 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.454755    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c"} err="failed to get container status \"ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c\": rpc error: code = NotFound desc = could not find container \"ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c\": container with ID starting with ae4c3f317192ef29d67a741ce382acd4006f9a0c67369d9e39029f8119bbd73c not found: ID does not exist"
Sep 18 20:04:52.454855 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.454776    2559 scope.go:117] "RemoveContainer" containerID="841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3"
Sep 18 20:04:52.455129 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:04:52.455105    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3\": container with ID starting with 841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3 not found: ID does not exist" containerID="841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3"
Sep 18 20:04:52.455190 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.455138    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3"} err="failed to get container status \"841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3\": rpc error: code = NotFound desc = could not find container \"841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3\": container with ID starting with 841c58d92332628f3d839e1734497478b4b91093937d8f24166ada4a97f1a1d3 not found: ID does not exist"
Sep 18 20:04:52.455190 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.455160    2559 scope.go:117] "RemoveContainer" containerID="6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01"
Sep 18 20:04:52.455468 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:04:52.455450    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01\": container with ID starting with 6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01 not found: ID does not exist" containerID="6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01"
Sep 18 20:04:52.455528 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.455477    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01"} err="failed to get container status \"6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01\": rpc error: code = NotFound desc = could not find container \"6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01\": container with ID starting with 6ad313e7b7542c4b2c2631c9eb2d38b437b41e0bdbb08c9f37052492e92f9a01 not found: ID does not exist"
Sep 18 20:04:52.455713 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:52.455693    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-c6f7b798f-g9c8p"]
Sep 18 20:04:53.470034 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:04:53.469999    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" path="/var/lib/kubelet/pods/7fa53863-2ece-4079-8590-ba1b442137c2/volumes"
Sep 18 20:05:01.815083 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.815048    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:01.848978 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.848932    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"]
Sep 18 20:05:01.849176 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849164    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="storage-initializer"
Sep 18 20:05:01.849232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849177    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="storage-initializer"
Sep 18 20:05:01.849232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849190    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="storage-initializer"
Sep 18 20:05:01.849232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849196    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="storage-initializer"
Sep 18 20:05:01.849232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849207    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container"
Sep 18 20:05:01.849232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849212    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container"
Sep 18 20:05:01.849232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849218    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container"
Sep 18 20:05:01.849232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849223    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container"
Sep 18 20:05:01.849232 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849231    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kube-rbac-proxy"
Sep 18 20:05:01.849465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849236    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kube-rbac-proxy"
Sep 18 20:05:01.849465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849244    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kube-rbac-proxy"
Sep 18 20:05:01.849465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849249    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kube-rbac-proxy"
Sep 18 20:05:01.849465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849287    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kserve-container"
Sep 18 20:05:01.849465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849295    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kserve-container"
Sep 18 20:05:01.849465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849300    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c248f96-4bea-454e-b5c9-8c144cfa7ee4" containerName="kube-rbac-proxy"
Sep 18 20:05:01.849465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.849306    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa53863-2ece-4079-8590-ba1b442137c2" containerName="kube-rbac-proxy"
Sep 18 20:05:01.853520 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.853504    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:01.855679 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.855662    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Sep 18 20:05:01.855679 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.855675    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 18 20:05:01.855829 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.855735    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 18 20:05:01.856265 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.856246    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9dxnf\""
Sep 18 20:05:01.856362 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.856328    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 18 20:05:01.862098 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.862080    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"]
Sep 18 20:05:01.893210 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:01.893181    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:02.038683 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.038658    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc478\" (UniqueName: \"kubernetes.io/projected/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-api-access-gc478\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.038780 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.038686    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.038780 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.038713    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38b9751-9f78-4e92-8f34-8592215a3d16-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.139628 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.139564    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc478\" (UniqueName: \"kubernetes.io/projected/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-api-access-gc478\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.139628 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.139591    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.139628 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.139615    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38b9751-9f78-4e92-8f34-8592215a3d16-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.140307 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.140279    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.142326 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.142304    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38b9751-9f78-4e92-8f34-8592215a3d16-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.147635 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.147603    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc478\" (UniqueName: \"kubernetes.io/projected/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-api-access-gc478\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-rqnx4\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.164501 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.164469    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:02.433341 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.433312    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"]
Sep 18 20:05:02.435772 ip-10-0-141-190 kubenswrapper[2559]: W0918 20:05:02.435742    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38b9751_9f78_4e92_8f34_8592215a3d16.slice/crio-2547ba92e1e1ed7cbc3c823eb0b7a0078840e0bd0883070d26dfb8cbcafcfeb6 WatchSource:0}: Error finding container 2547ba92e1e1ed7cbc3c823eb0b7a0078840e0bd0883070d26dfb8cbcafcfeb6: Status 404 returned error can't find the container with id 2547ba92e1e1ed7cbc3c823eb0b7a0078840e0bd0883070d26dfb8cbcafcfeb6
Sep 18 20:05:02.469172 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:02.469143    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" event={"ID":"d38b9751-9f78-4e92-8f34-8592215a3d16","Type":"ContainerStarted","Data":"2547ba92e1e1ed7cbc3c823eb0b7a0078840e0bd0883070d26dfb8cbcafcfeb6"}
Sep 18 20:05:03.477290 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:03.477253    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" event={"ID":"d38b9751-9f78-4e92-8f34-8592215a3d16","Type":"ContainerStarted","Data":"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874"}
Sep 18 20:05:03.477655 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:03.477304    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" event={"ID":"d38b9751-9f78-4e92-8f34-8592215a3d16","Type":"ContainerStarted","Data":"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e"}
Sep 18 20:05:03.477655 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:03.477400    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:03.494103 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:03.494053    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" podStartSLOduration=2.494034043 podStartE2EDuration="2.494034043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:03.492383507 +0000 UTC m=+1594.455116244" watchObservedRunningTime="2026-09-18 20:05:03.494034043 +0000 UTC m=+1594.456766780"
Sep 18 20:05:04.481276 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:04.481232    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:04.482986 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:04.482948    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:11.492890 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.492864    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:05:11.848465 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.848385    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:11.888702 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.888673    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"]
Sep 18 20:05:11.892046 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.892026    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:11.893812 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.893791    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 18 20:05:11.893977 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.893875    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 18 20:05:11.894060 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.894053    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 18 20:05:11.894130 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.894052    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 20:05:11.902674 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.902656    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"]
Sep 18 20:05:11.953744 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:11.953725    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:05:12.010506 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.010484    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.010574 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.010515    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.010574 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.010538    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.010642 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.010582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v75s5\" (UniqueName: \"kubernetes.io/projected/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-api-access-v75s5\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.010642 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.010629    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.111975 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.111902    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.111975 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.111938    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.112104 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.111975    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.112163 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.112104    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v75s5\" (UniqueName: \"kubernetes.io/projected/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-api-access-v75s5\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.112220 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.112169    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.112531 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.112511    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.112604 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.112588    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.112648 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.112626    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.114360 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.114342    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.119539 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.119519    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v75s5\" (UniqueName: \"kubernetes.io/projected/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-api-access-v75s5\") pod \"isvc-logger-raw-cipn-predictor-6b79574594-9swqs\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.203237 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.203220    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:12.472222 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.472197    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"]
Sep 18 20:05:12.474427 ip-10-0-141-190 kubenswrapper[2559]: W0918 20:05:12.474403    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde13c35_a2b1_4644_bda9_2e0bb5c223fe.slice/crio-02f7affdf291735c4c86ec97f509ec683ce6912d9911f91b643e8680c16e99d2 WatchSource:0}: Error finding container 02f7affdf291735c4c86ec97f509ec683ce6912d9911f91b643e8680c16e99d2: Status 404 returned error can't find the container with id 02f7affdf291735c4c86ec97f509ec683ce6912d9911f91b643e8680c16e99d2
Sep 18 20:05:12.514235 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:12.514210    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerStarted","Data":"02f7affdf291735c4c86ec97f509ec683ce6912d9911f91b643e8680c16e99d2"}
Sep 18 20:05:13.519092 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:13.519056    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerStarted","Data":"9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d"}
Sep 18 20:05:16.532798 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:16.532761    2559 generic.go:356] "Generic (PLEG): container finished" podID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerID="9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d" exitCode=0
Sep 18 20:05:16.533163 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:16.532837    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerDied","Data":"9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d"}
Sep 18 20:05:16.641840 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:16.641824    2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 20:05:17.540199 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:17.540162    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerStarted","Data":"2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b"}
Sep 18 20:05:17.540628 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:17.540245    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerStarted","Data":"9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a"}
Sep 18 20:05:19.551312 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:19.551272    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerStarted","Data":"f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5"}
Sep 18 20:05:19.551721 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:19.551508    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:19.551721 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:19.551583    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:19.552851 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:19.552824    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:05:19.570825 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:19.570759    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podStartSLOduration=2.370403469 podStartE2EDuration="8.570744435s" totalImagesPullingTime="2.443132361s" totalInitContainerRuntime="3.757208605s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:05:19.569412809 +0000 UTC m=+1610.532145549" watchObservedRunningTime="2026-09-18 20:05:19.570744435 +0000 UTC m=+1610.533477173"
Sep 18 20:05:20.555260 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:20.555229    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:20.555635 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:20.555371    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:05:20.556495 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:20.556466    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:05:21.559021 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:21.558948    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:05:21.559445 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:21.559321    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:05:21.562859 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:21.562838    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:05:22.566921 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:22.566878    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:05:22.567337 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:22.567210    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:05:32.567789 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:32.567745    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:05:32.568293 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:32.568267    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:05:42.567052 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:42.567010    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:05:42.567510 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:42.567417    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:05:52.567314 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:52.567277    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:05:52.567823 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:05:52.567798    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:06:02.567662 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:02.567617    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:06:02.568136 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:02.568065    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:06:12.567099 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:12.567055    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:06:12.567625 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:12.567602    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:06:22.567371 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:22.567328    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:06:22.567784 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:22.567713    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:06:32.568163 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:32.568126    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:06:32.568694 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:32.568311    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:06:46.960716 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:46.960644    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-rqnx4_d38b9751-9f78-4e92-8f34-8592215a3d16/kserve-container/0.log"
Sep 18 20:06:47.154585 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.154548    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"]
Sep 18 20:06:47.154900 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.154844    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" containerID="cri-o://9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a" gracePeriod=30
Sep 18 20:06:47.155098 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.154865    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" containerID="cri-o://f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5" gracePeriod=30
Sep 18 20:06:47.155098 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.154900    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy" containerID="cri-o://2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b" gracePeriod=30
Sep 18 20:06:47.184466 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.184445    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"]
Sep 18 20:06:47.184662 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.184631    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerName="kserve-container" containerID="cri-o://1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e" gracePeriod=30
Sep 18 20:06:47.184730 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.184656    2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerName="kube-rbac-proxy" containerID="cri-o://076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874" gracePeriod=30
Sep 18 20:06:47.188740 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.188648    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:47.245536 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.245476    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:47.428756 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.428731    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:06:47.466150 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.466127    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-rbac-proxy-sar-config\") pod \"d38b9751-9f78-4e92-8f34-8592215a3d16\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") "
Sep 18 20:06:47.466300 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.466159    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38b9751-9f78-4e92-8f34-8592215a3d16-proxy-tls\") pod \"d38b9751-9f78-4e92-8f34-8592215a3d16\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") "
Sep 18 20:06:47.466300 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.466183    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gc478\" (UniqueName: \"kubernetes.io/projected/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-api-access-gc478\") pod \"d38b9751-9f78-4e92-8f34-8592215a3d16\" (UID: \"d38b9751-9f78-4e92-8f34-8592215a3d16\") "
Sep 18 20:06:47.466481 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.466454    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d38b9751-9f78-4e92-8f34-8592215a3d16" (UID: "d38b9751-9f78-4e92-8f34-8592215a3d16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:06:47.468217 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.468195    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d38b9751-9f78-4e92-8f34-8592215a3d16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d38b9751-9f78-4e92-8f34-8592215a3d16" (UID: "d38b9751-9f78-4e92-8f34-8592215a3d16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:06:47.468289 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.468239    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-api-access-gc478" (OuterVolumeSpecName: "kube-api-access-gc478") pod "d38b9751-9f78-4e92-8f34-8592215a3d16" (UID: "d38b9751-9f78-4e92-8f34-8592215a3d16"). InnerVolumeSpecName "kube-api-access-gc478". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:06:47.567037 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.566985    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:06:47.567037 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.567007    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d38b9751-9f78-4e92-8f34-8592215a3d16-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:06:47.567037 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.567017    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc478\" (UniqueName: \"kubernetes.io/projected/d38b9751-9f78-4e92-8f34-8592215a3d16-kube-api-access-gc478\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:06:47.872400 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.872296    2559 generic.go:356] "Generic (PLEG): container finished" podID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerID="2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b" exitCode=2
Sep 18 20:06:47.872400 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.872365    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerDied","Data":"2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b"}
Sep 18 20:06:47.873937 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.873913    2559 generic.go:356] "Generic (PLEG): container finished" podID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerID="076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874" exitCode=2
Sep 18 20:06:47.873937 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.873934    2559 generic.go:356] "Generic (PLEG): container finished" podID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerID="1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e" exitCode=2
Sep 18 20:06:47.874128 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.873982    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"
Sep 18 20:06:47.874128 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.873995    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" event={"ID":"d38b9751-9f78-4e92-8f34-8592215a3d16","Type":"ContainerDied","Data":"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874"}
Sep 18 20:06:47.874128 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.874061    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" event={"ID":"d38b9751-9f78-4e92-8f34-8592215a3d16","Type":"ContainerDied","Data":"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e"}
Sep 18 20:06:47.874128 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.874083    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4" event={"ID":"d38b9751-9f78-4e92-8f34-8592215a3d16","Type":"ContainerDied","Data":"2547ba92e1e1ed7cbc3c823eb0b7a0078840e0bd0883070d26dfb8cbcafcfeb6"}
Sep 18 20:06:47.874128 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.874107    2559 scope.go:117] "RemoveContainer" containerID="076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874"
Sep 18 20:06:47.883225 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.883198    2559 scope.go:117] "RemoveContainer" containerID="1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e"
Sep 18 20:06:47.892633 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.892606    2559 scope.go:117] "RemoveContainer" containerID="076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874"
Sep 18 20:06:47.892721 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.892606    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"]
Sep 18 20:06:47.893044 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:06:47.893020    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874\": container with ID starting with 076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874 not found: ID does not exist" containerID="076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874"
Sep 18 20:06:47.893149 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.893057    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874"} err="failed to get container status \"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874\": rpc error: code = NotFound desc = could not find container \"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874\": container with ID starting with 076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874 not found: ID does not exist"
Sep 18 20:06:47.893149 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.893086    2559 scope.go:117] "RemoveContainer" containerID="1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e"
Sep 18 20:06:47.893566 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:06:47.893529    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e\": container with ID starting with 1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e not found: ID does not exist" containerID="1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e"
Sep 18 20:06:47.893566 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.893573    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e"} err="failed to get container status \"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e\": rpc error: code = NotFound desc = could not find container \"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e\": container with ID starting with 1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e not found: ID does not exist"
Sep 18 20:06:47.893566 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.893597    2559 scope.go:117] "RemoveContainer" containerID="076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874"
Sep 18 20:06:47.894061 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.894033    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874"} err="failed to get container status \"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874\": rpc error: code = NotFound desc = could not find container \"076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874\": container with ID starting with 076224015ab492ce59c939d7fa3d542322221ed3cd5be0bfba34944101fcd874 not found: ID does not exist"
Sep 18 20:06:47.894178 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.894077    2559 scope.go:117] "RemoveContainer" containerID="1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e"
Sep 18 20:06:47.894437 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.894419    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e"} err="failed to get container status \"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e\": rpc error: code = NotFound desc = could not find container \"1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e\": container with ID starting with 1f62e9d20c1a1ee36eb2da20311a7a3b0fca98c7af70942d57ebb5e40e864b0e not found: ID does not exist"
Sep 18 20:06:47.895328 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:47.895310    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-rqnx4"]
Sep 18 20:06:48.309637 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:48.309611    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:48.410299 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:48.410268    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:06:49.469494 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:49.469464    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" path="/var/lib/kubelet/pods/d38b9751-9f78-4e92-8f34-8592215a3d16/volumes"
Sep 18 20:06:51.559690 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:51.559656    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 20:06:51.889638 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:51.889566    2559 generic.go:356] "Generic (PLEG): container finished" podID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerID="9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a" exitCode=0
Sep 18 20:06:51.889638 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:51.889613    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerDied","Data":"9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a"}
Sep 18 20:06:52.567319 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:52.567287    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:06:52.567713 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:52.567648    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:06:56.559543 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:06:56.559508    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 20:07:01.559418 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:01.559377    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 20:07:01.559833 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:01.559507    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:07:02.566877 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:02.566832    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:07:02.567324 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:02.567166    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:07:03.904686 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:03.904656    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:03.927829 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:03.927799    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:03.937284 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:03.937264    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:03.947350 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:03.947321    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:04.779530 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:04.779499    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:04.792017 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:04.791982    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:04.801274 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:04.801256    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:04.811668 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:04.811647    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:05.630416 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:05.630384    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:05.642604 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:05.642569    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:05.651706 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:05.651685    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:05.662774 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:05.662743    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:06.469106 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:06.469068    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:06.482733 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:06.482709    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:06.492186 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:06.492162    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:06.502129 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:06.502112    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:06.559754 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:06.559717    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 20:07:07.304031 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:07.303999    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:07.316430 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:07.316408    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:07.326837 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:07.326793    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:07.337320 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:07.337297    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:08.133128 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:08.133097    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:08.145543 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:08.145517    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:08.155848 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:08.155822    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:08.167267 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:08.167244    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:09.069856 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:09.069830    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:09.082007 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:09.081979    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:09.091677 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:09.091653    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:09.101693 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:09.101668    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:09.913329 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:09.913299    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:09.925548 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:09.925522    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:09.935409 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:09.935375    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:09.945378 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:09.945355    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:10.750958 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:10.750926    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:10.762918 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:10.762893    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:10.772592 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:10.772569    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:10.783102 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:10.783080    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:11.559826 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:11.559789    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 20:07:11.574662 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:11.574635    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:11.586448 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:11.586421    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:11.597299 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:11.597275    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:11.607250 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:11.607227    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:12.424406 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:12.424371    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:12.436545 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:12.436518    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:12.446366 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:12.446346    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:12.457654 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:12.457639    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:12.567030 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:12.566993    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 18 20:07:12.567182 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:12.567166    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:07:12.567382 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:12.567355    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 20:07:12.567468 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:12.567457    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:07:13.275977 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:13.275948    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:13.288260 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:13.288225    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:13.298401 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:13.298380    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:13.308531 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:13.308514    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:14.122155 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:14.122128    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:14.135744 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:14.135717    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:14.147558 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:14.147529    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:14.159881 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:14.159862    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:15.070550 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:15.070511    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kserve-container/0.log"
Sep 18 20:07:15.084524 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:15.084499    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/kube-rbac-proxy/0.log"
Sep 18 20:07:15.096527 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:15.096501    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/agent/0.log"
Sep 18 20:07:15.109082 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:15.109061    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-6b79574594-9swqs_cde13c35-a2b1-4644-bda9-2e0bb5c223fe/storage-initializer/0.log"
Sep 18 20:07:16.559235 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:16.559189    2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 18 20:07:17.322782 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.322759    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:07:17.379532 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.379506    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kserve-provision-location\") pod \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") "
Sep 18 20:07:17.379653 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.379549    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v75s5\" (UniqueName: \"kubernetes.io/projected/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-api-access-v75s5\") pod \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") "
Sep 18 20:07:17.379653 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.379603    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-cabundle-cert\") pod \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") "
Sep 18 20:07:17.379653 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.379620    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-proxy-tls\") pod \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") "
Sep 18 20:07:17.379653 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.379646    2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-rbac-proxy-sar-config\") pod \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\" (UID: \"cde13c35-a2b1-4644-bda9-2e0bb5c223fe\") "
Sep 18 20:07:17.379838 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.379795    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cde13c35-a2b1-4644-bda9-2e0bb5c223fe" (UID: "cde13c35-a2b1-4644-bda9-2e0bb5c223fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 20:07:17.380055 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.380031    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cde13c35-a2b1-4644-bda9-2e0bb5c223fe" (UID: "cde13c35-a2b1-4644-bda9-2e0bb5c223fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:17.380139 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.380072    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cde13c35-a2b1-4644-bda9-2e0bb5c223fe" (UID: "cde13c35-a2b1-4644-bda9-2e0bb5c223fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 20:07:17.381633 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.381607    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cde13c35-a2b1-4644-bda9-2e0bb5c223fe" (UID: "cde13c35-a2b1-4644-bda9-2e0bb5c223fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 20:07:17.381734 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.381642    2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-api-access-v75s5" (OuterVolumeSpecName: "kube-api-access-v75s5") pod "cde13c35-a2b1-4644-bda9-2e0bb5c223fe" (UID: "cde13c35-a2b1-4644-bda9-2e0bb5c223fe"). InnerVolumeSpecName "kube-api-access-v75s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 20:07:17.480929 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.480911    2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-cabundle-cert\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:07:17.480929 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.480930    2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-proxy-tls\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:07:17.481068 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.480939    2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:07:17.481068 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.480948    2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kserve-provision-location\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:07:17.481068 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.480956    2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v75s5\" (UniqueName: \"kubernetes.io/projected/cde13c35-a2b1-4644-bda9-2e0bb5c223fe-kube-api-access-v75s5\") on node \"ip-10-0-141-190.ec2.internal\" DevicePath \"\""
Sep 18 20:07:17.985477 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.985445    2559 generic.go:356] "Generic (PLEG): container finished" podID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerID="f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5" exitCode=137
Sep 18 20:07:17.985884 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.985523    2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"
Sep 18 20:07:17.985884 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.985525    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerDied","Data":"f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5"}
Sep 18 20:07:17.985884 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.985593    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs" event={"ID":"cde13c35-a2b1-4644-bda9-2e0bb5c223fe","Type":"ContainerDied","Data":"02f7affdf291735c4c86ec97f509ec683ce6912d9911f91b643e8680c16e99d2"}
Sep 18 20:07:17.985884 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.985625    2559 scope.go:117] "RemoveContainer" containerID="f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5"
Sep 18 20:07:17.994074 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:17.994053    2559 scope.go:117] "RemoveContainer" containerID="2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b"
Sep 18 20:07:18.001061 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.001044    2559 scope.go:117] "RemoveContainer" containerID="9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a"
Sep 18 20:07:18.005004 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.004982    2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"]
Sep 18 20:07:18.008749 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.008730    2559 scope.go:117] "RemoveContainer" containerID="9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d"
Sep 18 20:07:18.010134 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.010112    2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-6b79574594-9swqs"]
Sep 18 20:07:18.015333 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.015319    2559 scope.go:117] "RemoveContainer" containerID="f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5"
Sep 18 20:07:18.015592 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:07:18.015574    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5\": container with ID starting with f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5 not found: ID does not exist" containerID="f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5"
Sep 18 20:07:18.015657 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.015602    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5"} err="failed to get container status \"f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5\": rpc error: code = NotFound desc = could not find container \"f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5\": container with ID starting with f0703608cc78376c751d4c182418b875e5e058ce0c021cd01431bc403d1838c5 not found: ID does not exist"
Sep 18 20:07:18.015657 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.015638    2559 scope.go:117] "RemoveContainer" containerID="2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b"
Sep 18 20:07:18.015900 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:07:18.015883    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b\": container with ID starting with 2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b not found: ID does not exist" containerID="2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b"
Sep 18 20:07:18.015937 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.015907    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b"} err="failed to get container status \"2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b\": rpc error: code = NotFound desc = could not find container \"2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b\": container with ID starting with 2669d148b7d4ff6d22569e4b9354ec47c0fd30733c4aee374f345aeb25770a4b not found: ID does not exist"
Sep 18 20:07:18.015937 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.015922    2559 scope.go:117] "RemoveContainer" containerID="9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a"
Sep 18 20:07:18.016173 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:07:18.016156    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a\": container with ID starting with 9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a not found: ID does not exist" containerID="9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a"
Sep 18 20:07:18.016230 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.016179    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a"} err="failed to get container status \"9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a\": rpc error: code = NotFound desc = could not find container \"9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a\": container with ID starting with 9947d113f9309134b182095dc05d321cda270a51c4be30fc91f2eceed59f397a not found: ID does not exist"
Sep 18 20:07:18.016230 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.016194    2559 scope.go:117] "RemoveContainer" containerID="9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d"
Sep 18 20:07:18.016430 ip-10-0-141-190 kubenswrapper[2559]: E0918 20:07:18.016412    2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d\": container with ID starting with 9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d not found: ID does not exist" containerID="9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d"
Sep 18 20:07:18.016486 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:18.016439    2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d"} err="failed to get container status \"9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d\": rpc error: code = NotFound desc = could not find container \"9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d\": container with ID starting with 9bbc4c77caa7820985749c3491ec1606f5ce27cb9b7a220c627fbbf3fa96a74d not found: ID does not exist"
Sep 18 20:07:19.469618 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:19.469587    2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" path="/var/lib/kubelet/pods/cde13c35-a2b1-4644-bda9-2e0bb5c223fe/volumes"
Sep 18 20:07:19.931201 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:19.931125    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:20.081175 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081142    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rlcn6/must-gather-4sxcc"]
Sep 18 20:07:20.081390 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081379    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerName="kube-rbac-proxy"
Sep 18 20:07:20.081430 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081393    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerName="kube-rbac-proxy"
Sep 18 20:07:20.081430 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081405    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent"
Sep 18 20:07:20.081430 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081411    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent"
Sep 18 20:07:20.081430 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081426    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerName="kserve-container"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081431    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerName="kserve-container"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081438    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081443    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081466    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="storage-initializer"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081473    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="storage-initializer"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081480    2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081485    2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081521    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kserve-container"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081529    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="kube-rbac-proxy"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081535    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerName="kube-rbac-proxy"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081541    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38b9751-9f78-4e92-8f34-8592215a3d16" containerName="kserve-container"
Sep 18 20:07:20.081571 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.081547    2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde13c35-a2b1-4644-bda9-2e0bb5c223fe" containerName="agent"
Sep 18 20:07:20.086026 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.086006    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rlcn6/must-gather-4sxcc"
Sep 18 20:07:20.088206 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.088186    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rlcn6\"/\"kube-root-ca.crt\""
Sep 18 20:07:20.088299 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.088249    2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rlcn6\"/\"default-dockercfg-dr5wp\""
Sep 18 20:07:20.088358 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.088321    2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rlcn6\"/\"openshift-service-ca.crt\""
Sep 18 20:07:20.094173 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.094153    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rlcn6/must-gather-4sxcc"]
Sep 18 20:07:20.094634 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.094609    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:20.193508 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.193429    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:20.200438 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.200421    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp867\" (UniqueName: \"kubernetes.io/projected/940c6bde-67ae-40e1-8e6e-4041b4c12589-kube-api-access-cp867\") pod \"must-gather-4sxcc\" (UID: \"940c6bde-67ae-40e1-8e6e-4041b4c12589\") " pod="openshift-must-gather-rlcn6/must-gather-4sxcc"
Sep 18 20:07:20.200505 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.200451    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/940c6bde-67ae-40e1-8e6e-4041b4c12589-must-gather-output\") pod \"must-gather-4sxcc\" (UID: \"940c6bde-67ae-40e1-8e6e-4041b4c12589\") " pod="openshift-must-gather-rlcn6/must-gather-4sxcc"
Sep 18 20:07:20.301013 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.300992    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp867\" (UniqueName: \"kubernetes.io/projected/940c6bde-67ae-40e1-8e6e-4041b4c12589-kube-api-access-cp867\") pod \"must-gather-4sxcc\" (UID: \"940c6bde-67ae-40e1-8e6e-4041b4c12589\") " pod="openshift-must-gather-rlcn6/must-gather-4sxcc"
Sep 18 20:07:20.301137 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.301024    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/940c6bde-67ae-40e1-8e6e-4041b4c12589-must-gather-output\") pod \"must-gather-4sxcc\" (UID: \"940c6bde-67ae-40e1-8e6e-4041b4c12589\") " pod="openshift-must-gather-rlcn6/must-gather-4sxcc"
Sep 18 20:07:20.301306 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.301290    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/940c6bde-67ae-40e1-8e6e-4041b4c12589-must-gather-output\") pod \"must-gather-4sxcc\" (UID: \"940c6bde-67ae-40e1-8e6e-4041b4c12589\") " pod="openshift-must-gather-rlcn6/must-gather-4sxcc"
Sep 18 20:07:20.308748 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.308723    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp867\" (UniqueName: \"kubernetes.io/projected/940c6bde-67ae-40e1-8e6e-4041b4c12589-kube-api-access-cp867\") pod \"must-gather-4sxcc\" (UID: \"940c6bde-67ae-40e1-8e6e-4041b4c12589\") " pod="openshift-must-gather-rlcn6/must-gather-4sxcc"
Sep 18 20:07:20.395940 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.395906    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rlcn6/must-gather-4sxcc"
Sep 18 20:07:20.660567 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.660540    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rlcn6/must-gather-4sxcc"]
Sep 18 20:07:20.662991 ip-10-0-141-190 kubenswrapper[2559]: W0918 20:07:20.662955    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod940c6bde_67ae_40e1_8e6e_4041b4c12589.slice/crio-2b0e3cc5fc7435270fa15329d49188636f6f6d8dbe76f0c94e42f0409b9ade7c WatchSource:0}: Error finding container 2b0e3cc5fc7435270fa15329d49188636f6f6d8dbe76f0c94e42f0409b9ade7c: Status 404 returned error can't find the container with id 2b0e3cc5fc7435270fa15329d49188636f6f6d8dbe76f0c94e42f0409b9ade7c
Sep 18 20:07:20.999102 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:20.999069    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rlcn6/must-gather-4sxcc" event={"ID":"940c6bde-67ae-40e1-8e6e-4041b4c12589","Type":"ContainerStarted","Data":"2b0e3cc5fc7435270fa15329d49188636f6f6d8dbe76f0c94e42f0409b9ade7c"}
Sep 18 20:07:22.007857 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:22.007820    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rlcn6/must-gather-4sxcc" event={"ID":"940c6bde-67ae-40e1-8e6e-4041b4c12589","Type":"ContainerStarted","Data":"27971f2e002179b69decf9722739fd7e7176568cf3c5b83140de78d5d30df06d"}
Sep 18 20:07:22.008347 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:22.007872    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rlcn6/must-gather-4sxcc" event={"ID":"940c6bde-67ae-40e1-8e6e-4041b4c12589","Type":"ContainerStarted","Data":"2e21bdc7ee699e7ede9951fec84f8ed40303ab4eded01543bdd8b2c57c8cf8b4"}
Sep 18 20:07:22.023732 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:22.023620    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rlcn6/must-gather-4sxcc" podStartSLOduration=1.315638799 podStartE2EDuration="2.023598374s" totalImagesPullingTime="707.959575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 20:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:22.022012274 +0000 UTC m=+1732.984745014" watchObservedRunningTime="2026-09-18 20:07:22.023598374 +0000 UTC m=+1732.986331112"
Sep 18 20:07:22.697504 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:22.697465    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-26877_65111b60-974a-4cc8-8faa-73e99a98233a/global-pull-secret-syncer/0.log"
Sep 18 20:07:22.829052 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:22.829001    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log"
Sep 18 20:07:22.944641 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:22.944560    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-190.ec2.internal_c80dc006f68abf1cd999156c6067d05e/haproxy/0.log"
Sep 18 20:07:25.359762 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:25.359717    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:25.460826 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:25.460702    2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5bdnh_000387d9-d879-418d-a2ca-a80f5883f7a5/konnectivity-agent/0.log" limit=1024
Sep 18 20:07:27.185889 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:27.185861    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-whlb4_e5e51ef1-9886-44bc-be5a-c04428bbb52d/node-exporter/0.log"
Sep 18 20:07:27.216006 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:27.215974    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-whlb4_e5e51ef1-9886-44bc-be5a-c04428bbb52d/kube-rbac-proxy/0.log"
Sep 18 20:07:27.242678 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:27.242645    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-whlb4_e5e51ef1-9886-44bc-be5a-c04428bbb52d/init-textfile/0.log"
Sep 18 20:07:27.641606 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:27.641573    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f59bdcb78-mwxcj_431a45a4-acb9-495d-a904-3691e3a21c72/telemeter-client/0.log"
Sep 18 20:07:27.672254 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:27.672221    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f59bdcb78-mwxcj_431a45a4-acb9-495d-a904-3691e3a21c72/reload/0.log"
Sep 18 20:07:27.696793 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:27.696723    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f59bdcb78-mwxcj_431a45a4-acb9-495d-a904-3691e3a21c72/kube-rbac-proxy/0.log"
Sep 18 20:07:30.995375 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:30.995342    2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"]
Sep 18 20:07:31.001119 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.001094    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.011462 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.011417    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"]
Sep 18 20:07:31.092480 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.092438    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-lib-modules\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.092681 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.092526    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-proc\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.092681 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.092557    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-podres\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.092681 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.092582    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrmdh\" (UniqueName: \"kubernetes.io/projected/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-kube-api-access-lrmdh\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.092681 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.092618    2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-sys\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193263 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193228    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-lib-modules\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193456 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193281    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-proc\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193456 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193305    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-podres\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193456 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193328    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrmdh\" (UniqueName: \"kubernetes.io/projected/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-kube-api-access-lrmdh\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193456 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193360    2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-sys\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193456 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193391    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-proc\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193456 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193437    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-lib-modules\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193456 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193448    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-sys\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.193868 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.193522    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-podres\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.200748 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.200717    2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrmdh\" (UniqueName: \"kubernetes.io/projected/9f18f6d3-978d-4a6a-a865-7ba1c896d4ce-kube-api-access-lrmdh\") pod \"perf-node-gather-daemonset-d7q29\" (UID: \"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce\") " pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.316584 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.316508    2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:31.582854 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.582794    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g792g_873ecdc4-ef5b-430d-aa6a-d0216f3344aa/dns/0.log"
Sep 18 20:07:31.602916 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.602885    2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"]
Sep 18 20:07:31.605924 ip-10-0-141-190 kubenswrapper[2559]: W0918 20:07:31.605900    2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9f18f6d3_978d_4a6a_a865_7ba1c896d4ce.slice/crio-df3450baeb872497054fcd593ab25f0460b30912f9e60c5f0eea86b780342bcb WatchSource:0}: Error finding container df3450baeb872497054fcd593ab25f0460b30912f9e60c5f0eea86b780342bcb: Status 404 returned error can't find the container with id df3450baeb872497054fcd593ab25f0460b30912f9e60c5f0eea86b780342bcb
Sep 18 20:07:31.608747 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.608723    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g792g_873ecdc4-ef5b-430d-aa6a-d0216f3344aa/kube-rbac-proxy/0.log"
Sep 18 20:07:31.701784 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:31.701757    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hvl8z_d8ab3246-2a81-4ed1-8f73-2e44beaeba85/dns-node-resolver/0.log"
Sep 18 20:07:32.060303 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:32.060261    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29" event={"ID":"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce","Type":"ContainerStarted","Data":"191d286dc6f657689e12cd4aca3248a1c1677aa71c3d0a0a607d403e3a05705c"}
Sep 18 20:07:32.060779 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:32.060325    2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29" event={"ID":"9f18f6d3-978d-4a6a-a865-7ba1c896d4ce","Type":"ContainerStarted","Data":"df3450baeb872497054fcd593ab25f0460b30912f9e60c5f0eea86b780342bcb"}
Sep 18 20:07:32.060779 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:32.060379    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:32.075909 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:32.075866    2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29" podStartSLOduration=2.075851547 podStartE2EDuration="2.075851547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 20:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 20:07:32.074686762 +0000 UTC m=+1743.037419500" watchObservedRunningTime="2026-09-18 20:07:32.075851547 +0000 UTC m=+1743.038584315"
Sep 18 20:07:32.192628 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:32.192594    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8444bdf559-88d46_25630fa6-a5c9-41b5-a564-2a15445ba92f/registry/0.log"
Sep 18 20:07:32.244121 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:32.244079    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ccfk8_0be5902d-0c5c-4c36-b955-a0d97e5e61f5/node-ca/0.log"
Sep 18 20:07:33.417389 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:33.417361    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fpsxk_789fc76c-eb30-45d8-9e8f-a6ac1f6fb324/serve-healthcheck-canary/0.log"
Sep 18 20:07:33.780559 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:33.780534    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6x52b_560d1f47-6808-41b0-b1ac-f0efdd6cdd16/kube-rbac-proxy/0.log"
Sep 18 20:07:33.802707 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:33.802685    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6x52b_560d1f47-6808-41b0-b1ac-f0efdd6cdd16/exporter/0.log"
Sep 18 20:07:33.825511 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:33.825490    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6x52b_560d1f47-6808-41b0-b1ac-f0efdd6cdd16/extractor/0.log"
Sep 18 20:07:36.195676 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:36.195649    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-vk5jt_eb9123bf-09eb-4f12-94cf-f96097be22f6/server/0.log"
Sep 18 20:07:36.421134 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:36.421098    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-px9dz_a2fdb58e-4daa-405c-94e4-da7c75a3f12d/manager/0.log"
Sep 18 20:07:36.509906 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:36.509879    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-v5vb4_5da113bf-6f06-480e-9f1b-b48ea16f264f/seaweedfs/0.log"
Sep 18 20:07:38.076100 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:38.076070    2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rlcn6/perf-node-gather-daemonset-d7q29"
Sep 18 20:07:42.186706 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.186675    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89bjd_70099e14-8e28-4a33-b5e2-a29a25a1e226/kube-multus-additional-cni-plugins/0.log"
Sep 18 20:07:42.215551 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.215506    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89bjd_70099e14-8e28-4a33-b5e2-a29a25a1e226/egress-router-binary-copy/0.log"
Sep 18 20:07:42.240270 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.240205    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89bjd_70099e14-8e28-4a33-b5e2-a29a25a1e226/cni-plugins/0.log"
Sep 18 20:07:42.263487 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.263465    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89bjd_70099e14-8e28-4a33-b5e2-a29a25a1e226/bond-cni-plugin/0.log"
Sep 18 20:07:42.285759 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.285726    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89bjd_70099e14-8e28-4a33-b5e2-a29a25a1e226/routeoverride-cni/0.log"
Sep 18 20:07:42.307596 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.307573    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89bjd_70099e14-8e28-4a33-b5e2-a29a25a1e226/whereabouts-cni-bincopy/0.log"
Sep 18 20:07:42.330785 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.330764    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-89bjd_70099e14-8e28-4a33-b5e2-a29a25a1e226/whereabouts-cni/0.log"
Sep 18 20:07:42.374815 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.374757    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g6zjk_324f320a-8733-4798-a3a8-9541f5119139/kube-multus/0.log"
Sep 18 20:07:42.478317 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.478291    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vfx7n_d271732b-e2a8-4cc5-b5e6-6481da077fbb/network-metrics-daemon/0.log"
Sep 18 20:07:42.499429 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:42.499370    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vfx7n_d271732b-e2a8-4cc5-b5e6-6481da077fbb/kube-rbac-proxy/0.log"
Sep 18 20:07:43.893903 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:43.893867    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-controller/0.log"
Sep 18 20:07:43.928013 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:43.927983    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/1.log"
Sep 18 20:07:43.928173 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:43.928026    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovn-acl-logging/0.log"
Sep 18 20:07:43.957293 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:43.957267    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/kube-rbac-proxy-node/0.log"
Sep 18 20:07:43.984373 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:43.984349    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 20:07:44.002552 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:44.002529    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/northd/0.log"
Sep 18 20:07:44.024719 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:44.024700    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/nbdb/0.log"
Sep 18 20:07:44.048567 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:44.048549    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/sbdb/0.log"
Sep 18 20:07:44.107715 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:44.107695    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dt7qz_f0aed8e6-4720-49e3-96e7-2826644f38b7/ovnkube-controller/0.log"
Sep 18 20:07:45.104219 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:45.104189    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-86rp7_f744fa00-c636-4c25-a403-0635a8f15523/network-check-target-container/0.log"
Sep 18 20:07:46.008777 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:46.008751    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jhjbw_595bb485-5ce7-4268-97b9-df107f423e28/iptables-alerter/0.log"
Sep 18 20:07:46.643510 ip-10-0-141-190 kubenswrapper[2559]: I0918 20:07:46.643480    2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-hdqj7_3ca9d706-4a2b-4f4b-8eba-4b84e2f48b5f/tuned/0.log"