Sep 17 15:52:10.949704 ip-10-0-128-128 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 15:52:10.949723 ip-10-0-128-128 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 15:52:10.949734 ip-10-0-128-128 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 15:52:10.950175 ip-10-0-128-128 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 15:52:20.446153 ip-10-0-128-128 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 17 15:52:20.446169 ip-10-0-128-128 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2b5ece887c014f9c91f792fcc93bdf2d --
Sep 17 15:54:01.643130 ip-10-0-128-128 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:54:02.087340 ip-10-0-128-128 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:54:02.087340 ip-10-0-128-128 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:54:02.087340 ip-10-0-128-128 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:54:02.087340 ip-10-0-128-128 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:54:02.087340 ip-10-0-128-128 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:54:02.088555 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.088469    2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:54:02.097310 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097289    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:02.097310 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097308    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:02.097310 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097312    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097315    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097318    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097321    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097324    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097327    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097335    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097338    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097340    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097343    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097346    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097349    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097352    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097355    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097357    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097360    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097362    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097365    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097368    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097370    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:02.097405 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097373    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097376    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097378    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097381    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097384    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097386    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097389    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097392    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097394    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097397    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097399    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097402    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097404    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097407    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097410    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097413    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097416    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097419    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097422    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097425    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:02.097899 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097427    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097430    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097432    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097435    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097438    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097442    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097446    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097449    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097452    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097454    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097458    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097461    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097465    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097468    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097470    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097473    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097476    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097478    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097481    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:02.098383 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097484    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097486    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097489    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097491    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097494    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097496    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097500    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097503    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097506    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097510    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097514    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097517    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097520    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097523    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097526    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097528    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097530    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097533    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097536    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097538    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:02.098853 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097540    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097545    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097548    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097551    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097554    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097556    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097559    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097562    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097564    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097567    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097569    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097572    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097574    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097577    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097579    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097582    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097585    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.097587    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098303    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:02.099378 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098310    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098313    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098316    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098318    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098321    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098324    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098326    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098329    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098332    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098335    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098338    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098340    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098342    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098345    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098348    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098351    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098353    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098356    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098358    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:02.099858 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098362    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098366    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098369    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098372    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098375    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098378    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098381    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098383    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098386    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098389    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098392    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098395    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098398    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098401    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098404    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098407    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098410    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098413    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098416    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098419    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:02.100331 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098422    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098425    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098428    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098430    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098433    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098436    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098439    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098441    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098444    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098446    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098449    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098452    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098454    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098457    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098459    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098462    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098464    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098467    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098469    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098472    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:02.100851 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098474    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098477    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098480    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098482    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098485    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098491    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098493    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098496    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098499    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098503    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098506    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098508    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098511    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098513    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098517    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098519    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098522    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098524    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098527    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:02.101356 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098529    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098532    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098534    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098537    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098539    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098542    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098544    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098547    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098549    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098552    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098554    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098557    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098559    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098563    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098567    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098570    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098573    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098576    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098579    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:02.101825 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098582    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098973    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098980    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098983    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098988    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098992    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098995    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.098998    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099002    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099005    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099008    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099011    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099013    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099016    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099019    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099022    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099024    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099027    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099030    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:02.102297 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099032    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099035    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099037    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099040    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099043    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099046    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099048    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099051    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099054    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099056    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099059    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099064    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099067    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099069    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099072    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099074    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099077    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099080    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099082    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099085    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:02.102776 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099087    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099090    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099093    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099095    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099098    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099101    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099103    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099106    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099109    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099111    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099114    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099116    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099119    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099121    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099124    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099127    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099129    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099132    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099135    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099137    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:02.103270 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099140    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099143    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099145    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099149    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099152    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099155    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099157    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099160    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099163    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099165    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099168    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099170    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099173    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099175    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099178    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099181    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099183    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099186    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099189    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099191    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:02.103769 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099194    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099196    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099199    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099201    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099204    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099209    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099212    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099215    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099218    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099221    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099224    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099227    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099230    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099233    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099236    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099240    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099242    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099245    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099248    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:02.104387 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099251    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.099253    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099331    2584 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099364    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099370    2584 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099374    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099378    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099382    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099386    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099392    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099395    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099398    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099402    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099405    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099408    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099411    2584 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099414    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099417    2584 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099420    2584 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099430    2584 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099434    2584 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099437    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099440    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099443    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:54:02.104907 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099447    2584 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099451    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099454    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099457    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099462    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099465    2584 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099468    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099471    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099474    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099477    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099481    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099484    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099487    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099490    2584 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099493    2584 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099496    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099500    2584 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099503    2584 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099515    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099518    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099521    2584 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099525    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099528    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099531    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099534    2584 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:54:02.105501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099537    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099540    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099544    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099547    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099550    2584 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099553    2584 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099556    2584 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099559    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099562    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099566    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099569    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099574    2584 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099577    2584 flags.go:64] FLAG: --help="false"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099580    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099583    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099586    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099589    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099592    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099595    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099598    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099601    2584 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099604    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099607    2584 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:54:02.106108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099610    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099613    2584 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099616    2584 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099619    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099622    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099625    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099628    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099630    2584 flags.go:64] FLAG: --lock-file=""
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099633    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099636    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099639    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099646    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099649    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099652    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099655    2584 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099658    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099662    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099664    2584 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099667    2584 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099672    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099676    2584 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099680    2584 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099683    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099686    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099689    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:54:02.106685 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099692    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099695    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099697    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099700    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099713    2584 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099716    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099719    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099722    2584 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099725    2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099731    2584 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099734    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099737    2584 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099741    2584 flags.go:64] FLAG: --port="10250"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099744    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099747    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d5dd9b2c1178488"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099750    2584 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099753    2584 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099756    2584 flags.go:64] FLAG: --register-node="true"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099760    2584 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099763    2584 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099766    2584 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099769    2584 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099772    2584 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099776    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099792    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:54:02.107312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099797    2584 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099802    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099805    2584 flags.go:64] FLAG: --runonce="false"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099810    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099813    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099816    2584 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099819    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099822    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099825    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099828    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099832    2584 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099834    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099837    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099840    2584 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099843    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099846    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099850    2584 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099852    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099858    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099862    2584 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099865    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099869    2584 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099872    2584 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099875    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099879    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:54:02.108012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099883    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099886    2584 flags.go:64] FLAG: --v="2"
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099890    2584 flags.go:64] FLAG: --version="false"
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099894    2584 flags.go:64] FLAG: --vmodule=""
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099897    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.099901    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100037    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100042    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100045    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100048    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100053    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100055    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100059    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100061    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100064    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100066    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100071    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100074    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100077    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100080    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100083    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:02.108645 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100086    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100089    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100091    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100094    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100097    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100099    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100103    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100105    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100108    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100111    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100113    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100117    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100120    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100122    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100125    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100128    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100130    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100133    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100135    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100138    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:02.109171 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100141    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100144    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100147    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100150    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100153    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100155    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100158    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100161    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100163    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100166    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100168    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100171    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100175    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100179    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100182    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100185    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100187    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100190    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100193    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:02.109819 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100201    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100204    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100207    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100209    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100213    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100215    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100218    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100221    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100223    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100226    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100229    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100231    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100234    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100236    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100240    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100243    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100245    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100248    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100250    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100253    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:02.110519 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100256    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100258    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100261    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100264    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100266    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100269    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100271    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100274    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100277    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100279    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100282    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100284    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100287    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100297    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100301    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100304    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100308    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100311    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100314    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:02.111071 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100316    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:02.111586 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100319    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:02.111586 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100322    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:02.111586 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100324    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:02.111586 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100327    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:02.111586 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.100330    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:02.111586 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.101231    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:54:02.111586 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.110928    2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:54:02.111586 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.110943    2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:54:02.111845 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.111833    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:54:02.111882 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.111845    2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:54:02.111917 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111909    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:02.111917 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111917    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111922    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111926    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111929    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111933    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111936    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111938    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111941    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111944    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111947    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111949    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111952    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111955    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111957    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111960    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111963    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111965    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111968    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111971    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:02.111973 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111973    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111976    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111979    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111982    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111985    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111987    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111991    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111994    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111996    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.111999    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112002    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112004    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112007    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112010    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112013    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112015    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112018    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112021    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112023    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112026    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:02.112441 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112029    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112032    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112034    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112037    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112039    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112042    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112046    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112049    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112052    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112054    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112057    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112059    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112062    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112064    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112069    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112073    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112076    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112079    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112082    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112085    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:02.112955 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112088    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112090    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112093    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112097    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112101    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112104    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112107    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112109    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112113    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112115    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112118    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112121    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112123    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112126    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112128    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112131    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112134    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112136    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112139    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112142    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:02.113447 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112145    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112148    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112150    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112153    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112156    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112158    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112161    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112164    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112167    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112169    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112172    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112175    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112177    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112180    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112183    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112185    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112188    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112191    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:02.113946 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112193    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.112199    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112318    2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112323    2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112326    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112329    2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112332    2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112334    2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112337    2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112340    2584 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112342    2584 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112346    2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112351    2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112354    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112358    2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:54:02.114384 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112361    2584 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112364    2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112367    2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112371    2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112374    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112377    2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112380    2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112382    2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112385    2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112388    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112390    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112393    2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112395    2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112398    2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112401    2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112404    2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112407    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112409    2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112412    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:54:02.114760 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112414    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112417    2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112419    2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112422    2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112425    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112427    2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112430    2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112432    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112435    2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112437    2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112440    2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112443    2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112446    2584 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112449    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112452    2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112454    2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112457    2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112459    2584 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112462    2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112465    2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:54:02.115226 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112468    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112470    2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112473    2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112475    2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112478    2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112481    2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112483    2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112487    2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112489    2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112492    2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112495    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112497    2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112500    2584 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112502    2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112505    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112508    2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112510    2584 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112513    2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112515    2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112518    2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:54:02.115722 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112520    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112523    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112525    2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112528    2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112530    2584 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112533    2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112536    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112539    2584 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112542    2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112546    2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112550    2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112552    2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112555    2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112557    2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112560    2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112562    2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112565    2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112567    2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112570    2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112572    2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:54:02.116228 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112575    2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112578    2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112580    2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112583    2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112585    2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112588    2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:02.112591    2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.112596    2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.112712    2584 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:54:02.116709 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.116197    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:54:02.117284 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.117273    2584 server.go:1019] "Starting client certificate rotation"
Sep 17 15:54:02.117396 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.117378    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:54:02.117426 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.117416    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:54:02.120114 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.120100    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:54:02.125003 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.124990    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:54:02.133807 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.133774    2584 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:54:02.135842 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.135826    2584 log.go:25] "Validated CRI v1 image API"
Sep 17 15:54:02.136706 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.136689    2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:54:02.139585 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.139556    2584 fs.go:135] Filesystem UUIDs: map[2cb1e685-c50e-4f61-b8c4-eceb9b9aceb8:/dev/nvme0n1p3 651f908a-6cf1-49ac-a238-d0e891fa38bc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 15:54:02.139661 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.139583    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:54:02.145545 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.145415    2584 manager.go:216] Machine: {Timestamp:2026-09-17 15:54:02.143960685 +0000 UTC m=+0.370629984 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100437 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27bc6aa5d901c87837e152a44f5c44 SystemUUID:ec27bc6a-a5d9-01c8-7837-e152a44f5c44 BootID:2b5ece88-7c01-4f9c-91f7-92fcc93bdf2d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:22:ef:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:22:ef:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:2e:27:40:11:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:54:02.145545 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.145533    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:54:02.145693 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.145628    2584 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:54:02.149053 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.149018    2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:54:02.149202 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.149055    2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:54:02.149926 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.149915    2584 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:54:02.149958 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.149929    2584 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:54:02.149958 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.149937    2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:54:02.149958 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.149952    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:54:02.150043 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.150029    2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:54:02.154715 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.154700    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:54:02.154872 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.154860    2584 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:54:02.154914 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.154894    2584 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:54:02.154914 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.154904    2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:54:02.154974 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.154915    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:54:02.154974 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.154925    2584 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:54:02.154974 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.154933    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:54:02.156314 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.156286    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:54:02.156418 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.156328    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:54:02.158970 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.158945    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:54:02.159257 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.159238    2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 15:54:02.162287 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.162273    2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:54:02.162332 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.162303    2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:54:02.163805 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163776    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:54:02.163858 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163814    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:54:02.163858 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163824    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:54:02.163858 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163832    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:54:02.163858 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163844    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:54:02.163858 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163852    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:54:02.163983 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163870    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:54:02.163983 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163878    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:54:02.163983 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163888    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:54:02.163983 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163896    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:54:02.163983 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163908    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:54:02.163983 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.163916    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:54:02.164840 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.164829    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:54:02.164871 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.164841    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:54:02.168905 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.168891    2584 server.go:1268] "Started kubelet"
Sep 17 15:54:02.168973 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.168943    2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:54:02.169090 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.169048    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:54:02.169139 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.169118    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:54:02.169768 ip-10-0-128-128 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:54:02.171024 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.170923    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:54:02.171175 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.171154    2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:54:02.171838 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.171774    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:54:02.171838 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.171774    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:54:02.173896 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.173882    2584 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:54:02.176231 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.176212    2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:54:02.178071 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.176942    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d626b0ac0b279a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-17 15:54:02.168862618 +0000 UTC m=+0.395531898,LastTimestamp:2026-09-17 15:54:02.168862618 +0000 UTC m=+0.395531898,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 17 15:54:02.181867 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.181848    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:54:02.182620 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.182601    2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:54:02.182830 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.182811    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:02.183441 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.183378    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:54:02.183496 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.183474    2584 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:54:02.183496 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.183486    2584 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:54:02.183592 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.183577    2584 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:54:02.183592 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.183588    2584 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:54:02.184520 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.184503    2584 factory.go:55] Registering systemd factory
Sep 17 15:54:02.184603 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.184526    2584 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:54:02.184812 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.184797    2584 factory.go:153] Registering CRI-O factory
Sep 17 15:54:02.184857 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.184815    2584 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:54:02.184893 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.184857    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:54:02.184893 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.184875    2584 factory.go:103] Registering Raw factory
Sep 17 15:54:02.184970 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.184910    2584 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:54:02.185435 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.185422    2584 manager.go:349] Starting recovery of all containers
Sep 17 15:54:02.185893 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.185797    2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:54:02.186079 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.186052    2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:54:02.193954 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.193903    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:54:02.197319 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.197303    2584 manager.go:354] Recovery completed
Sep 17 15:54:02.202773 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.202759    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:02.204924 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.204898    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:02.204994 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.204936    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:02.204994 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.204946    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:02.205440 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.205425    2584 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:54:02.205440 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.205439    2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:54:02.205589 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.205448    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:02.205589 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.205457    2584 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:54:02.206047 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.206031    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:02.206096 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.206063    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:02.206096 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.206078    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:02.207668 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.207654    2584 policy_none.go:49] "None policy: Start"
Sep 17 15:54:02.207730 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.207672    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:54:02.207730 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.207685    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:54:02.209705 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.209693    2584 policy_none.go:47] "Start"
Sep 17 15:54:02.218946 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.218774    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d626b0ae31612c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-17 15:54:02.204922156 +0000 UTC m=+0.431591435,LastTimestamp:2026-09-17 15:54:02.204922156 +0000 UTC m=+0.431591435,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 17 15:54:02.228670 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.228594    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d626b0ae31abac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-128.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-17 15:54:02.204941228 +0000 UTC m=+0.431610507,LastTimestamp:2026-09-17 15:54:02.204941228 +0000 UTC m=+0.431610507,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 17 15:54:02.242489 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.242466    2584 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:54:02.242605 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.242531    2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:54:02.242605 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.242545    2584 server.go:87] "Starting device plugin registration server"
Sep 17 15:54:02.242860 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.242844    2584 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:54:02.242943 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.242860    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:54:02.242991 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.242947    2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:54:02.243040 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.243031    2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:54:02.243082 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.243041    2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:54:02.243762 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.243744    2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:54:02.243922 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.243810    2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:02.244057 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.243973    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d626b0ae31d2bc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-128.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-17 15:54:02.204951228 +0000 UTC m=+0.431620509,LastTimestamp:2026-09-17 15:54:02.204951228 +0000 UTC m=+0.431620509,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 17 15:54:02.256690 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.256663    2584 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2623 into the system container \"/system.slice\": <nil>"
Sep 17 15:54:02.258852 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.258771    2584 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-128.ec2.internal.18d626b0ae31612c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d626b0ae31612c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-17 15:54:02.204922156 +0000 UTC m=+0.431591435,LastTimestamp:2026-09-17 15:54:02.20604751 +0000 UTC m=+0.432716800,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 17 15:54:02.273575 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.273500    2584 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-128.ec2.internal.18d626b0ae31abac\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d626b0ae31abac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-128.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-17 15:54:02.204941228 +0000 UTC m=+0.431610507,LastTimestamp:2026-09-17 15:54:02.206069033 +0000 UTC m=+0.432738313,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 17 15:54:02.283997 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.283917    2584 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-128.ec2.internal.18d626b0ae31d2bc\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d626b0ae31d2bc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-128.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-17 15:54:02.204951228 +0000 UTC m=+0.431620509,LastTimestamp:2026-09-17 15:54:02.206083814 +0000 UTC m=+0.432753093,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 17 15:54:02.294371 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.294300    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-128.ec2.internal.18d626b0b09067d1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-128.ec2.internal,UID:ip-10-0-128-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-128-128.ec2.internal,},FirstTimestamp:2026-09-17 15:54:02.244704209 +0000 UTC m=+0.471373480,LastTimestamp:2026-09-17 15:54:02.244704209 +0000 UTC m=+0.471373480,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-128.ec2.internal,}"
Sep 17 15:54:02.307747 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.307721    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9djw8"
Sep 17 15:54:02.325220 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.325194    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9djw8"
Sep 17 15:54:02.343243 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.343164    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:02.344207 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.344188    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:02.344335 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.344224    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:02.344335 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.344236    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:02.344335 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.344262    2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.365225 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.365203    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:54:02.369274 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.365255    2584 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:54:02.369274 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.365279    2584 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:54:02.369274 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.365379    2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:54:02.369274 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.368943    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:02.373552 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.373533    2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.373635 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.373557    2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-128.ec2.internal\": node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:02.448630 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.448599    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:02.466274 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.466238    2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"]
Sep 17 15:54:02.466386 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.466319    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:02.468034 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.468015    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:02.468132 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.468045    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:02.468132 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.468055    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:02.470495 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.470481    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:02.471009 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.470953    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.471115 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.471044    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:02.472905 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.472889    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:02.472994 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.472916    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:02.472994 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.472931    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:02.472994 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.472951    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:02.472994 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.472975    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:02.472994 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.472986    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:02.475273 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.475258    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.475367 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.475289    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:02.476013 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.476000    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:02.476090 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.476023    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:02.476090 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.476033    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:02.484796 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.484761    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cddee824a28fdff15b30d083c82658d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"cddee824a28fdff15b30d083c82658d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.484876 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.484811    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cddee824a28fdff15b30d083c82658d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"cddee824a28fdff15b30d083c82658d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.484876 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.484852    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42f9cee7deb9c35203a248ac01b72173-config\") pod \"kube-apiserver-proxy-ip-10-0-128-128.ec2.internal\" (UID: \"42f9cee7deb9c35203a248ac01b72173\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.491466 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.491448    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-128.ec2.internal\" not found" node="ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.496069 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.496050    2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-128.ec2.internal\" not found" node="ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.549514 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.549481    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:02.585373 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.585338    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cddee824a28fdff15b30d083c82658d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"cddee824a28fdff15b30d083c82658d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.585373 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.585371    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cddee824a28fdff15b30d083c82658d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"cddee824a28fdff15b30d083c82658d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.585552 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.585390    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42f9cee7deb9c35203a248ac01b72173-config\") pod \"kube-apiserver-proxy-ip-10-0-128-128.ec2.internal\" (UID: \"42f9cee7deb9c35203a248ac01b72173\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.585552 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.585447    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cddee824a28fdff15b30d083c82658d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"cddee824a28fdff15b30d083c82658d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.585552 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.585460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/42f9cee7deb9c35203a248ac01b72173-config\") pod \"kube-apiserver-proxy-ip-10-0-128-128.ec2.internal\" (UID: \"42f9cee7deb9c35203a248ac01b72173\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.585552 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.585466    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cddee824a28fdff15b30d083c82658d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal\" (UID: \"cddee824a28fdff15b30d083c82658d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.650464 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.650385    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:02.751461 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.751429    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:02.796828 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.796796    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.801241 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:02.801219    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:02.851768 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.851727    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:02.957371 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:02.957265    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.058024 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:03.057972    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.068446 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.068417    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:03.117938 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.117898    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:54:03.118612 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.118078    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:54:03.118612 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.118101    2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:54:03.159346 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:03.159308    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.259980 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:03.259911    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.264624 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.264604    2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:03.338951 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.338923    2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:54:03.342895 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.342867    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:54:03.342895 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.342905    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:54:03.343133 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.342916    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:54:03.351476 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.351440    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:49:03 +0000 UTC" deadline="2028-03-20 12:30:48.138044919 +0000 UTC"
Sep 17 15:54:03.351476 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.351470    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13196h36m44.786578662s"
Sep 17 15:54:03.359989 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:03.359964    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.418669 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:03.418629    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddee824a28fdff15b30d083c82658d1.slice/crio-b4f1d4328d7f8c8d98ab8f4991714962855ecb9913d2e6a50412e42c7feeb283 WatchSource:0}: Error finding container b4f1d4328d7f8c8d98ab8f4991714962855ecb9913d2e6a50412e42c7feeb283: Status 404 returned error can't find the container with id b4f1d4328d7f8c8d98ab8f4991714962855ecb9913d2e6a50412e42c7feeb283
Sep 17 15:54:03.424162 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:03.424139    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f9cee7deb9c35203a248ac01b72173.slice/crio-7d4db4a6589f2b01982e1b68e3b87c17ccca288237f53d98f4fab0e239af6466 WatchSource:0}: Error finding container 7d4db4a6589f2b01982e1b68e3b87c17ccca288237f53d98f4fab0e239af6466: Status 404 returned error can't find the container with id 7d4db4a6589f2b01982e1b68e3b87c17ccca288237f53d98f4fab0e239af6466
Sep 17 15:54:03.425086 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.425073    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:54:03.460933 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:03.460889    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.561177 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:03.561094    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.662200 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:03.662176    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.766245 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:03.766208    2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-128.ec2.internal\" not found"
Sep 17 15:54:03.772172 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.772143    2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:03.783807 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.783769    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:03.807981 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.807955    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:54:03.809236 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.809220    2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal"
Sep 17 15:54:03.825597 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:03.825577    2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:54:04.159440 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.159356    2584 apiserver.go:52] "Watching apiserver"
Sep 17 15:54:04.172744 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.172253    2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:54:04.174898 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.174489    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal","openshift-multus/multus-fqvct","openshift-network-diagnostics/network-check-target-h9dg9","openshift-ovn-kubernetes/ovnkube-node-szc22","kube-system/global-pull-secret-syncer-w8xbr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c","openshift-cluster-node-tuning-operator/tuned-t95wm","openshift-dns/node-resolver-n4hh7","openshift-image-registry/node-ca-g82l6","openshift-multus/multus-additional-cni-plugins-kxc2j","openshift-multus/network-metrics-daemon-wn4lc","openshift-network-operator/iptables-alerter-8bqtq","kube-system/konnectivity-agent-c8x87"]
Sep 17 15:54:04.177426 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.177339    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.179848 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.179633    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.183908 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.182276    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.183908 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.182986    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:54:04.186055 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.186036    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:04.186391 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.186361    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:04.191582 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.190743    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.193817 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.193792    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.193921 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.193893    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:04.194143 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194115    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-hostroot\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.194143 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194126    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:54:04.194301 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-socket-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.194301 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d594aab7-c878-4797-b9a3-2e04a55a1586-cni-binary-copy\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.194301 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194208    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.194301 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194241    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-registration-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.194301 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-system-cni-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194313    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194324    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-cni-bin\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194348    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194126    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194352    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-k8s-cni-cncf-io\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194472    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2kgcg\""
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194494    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-multus-certs\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194533    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7l7\" (UniqueName: \"kubernetes.io/projected/a8e9ef4a-e99c-4243-bd91-fbe366802378-kube-api-access-rq7l7\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.194567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-iptables-alerter-script\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194579    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194595    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz8tv\" (UniqueName: \"kubernetes.io/projected/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-kube-api-access-zz8tv\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194621    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-kubelet\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-conf-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194679    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-sys-fs\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194691    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194703    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-cni-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194728    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-netns\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194751    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-device-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194790    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194820    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194932    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-46b8l\""
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.194775    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.195036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-cni-multus\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195069    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-etc-kubernetes\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195112    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-host-slash\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195138    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-daemon-config\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195184    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6kb\" (UniqueName: \"kubernetes.io/projected/d594aab7-c878-4797-b9a3-2e04a55a1586-kube-api-access-rt6kb\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195223    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-cnibin\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195287    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-socket-dir-parent\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195263    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5vlgb\""
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195316    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195337    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rb7vs\""
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195370    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195395    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:54:04.195737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.195604    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:54:04.196385 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.196207    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.198273 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.198249    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:54:04.198641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.198623    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:04.201021 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.201000    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:54:04.202396 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.202064    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.202893 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.202873    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:54:04.205153 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.205135    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.207935 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.207574    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.210182 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.210067    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:04.210182 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.210142    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:04.213747 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.213727    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:54:04.213983 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.213963    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:54:04.214065 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.214013    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:54:04.214241 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.214222    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:54:04.214474 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.214453    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:54:04.214582 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.214236    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:54:04.214980 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.214943    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wjkhn\""
Sep 17 15:54:04.215117 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.214721    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:54:04.215183 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.215160    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:54:04.215873 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.215521    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:54:04.215873 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.215564    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:54:04.215873 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.215736    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-djljt\""
Sep 17 15:54:04.215873 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.215752    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:54:04.215873 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.215809    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lv2l7\""
Sep 17 15:54:04.216131 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.216079    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:54:04.216174 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.216134    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v4lv2\""
Sep 17 15:54:04.216174 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.216159    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f898t\""
Sep 17 15:54:04.237555 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.237534    2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:54:04.285426 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.285394    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:54:04.295617 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295585    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovnkube-config\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.295760 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295630    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42747b31-db23-4a26-85cd-ccd997d13d0c-konnectivity-ca\") pod \"konnectivity-agent-c8x87\" (UID: \"42747b31-db23-4a26-85cd-ccd997d13d0c\") " pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:04.295760 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295661    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-system-cni-dir\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.295760 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.295760 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295707    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcbvl\" (UniqueName: \"kubernetes.io/projected/fdbe84e1-dd8f-4701-93af-2087896382b3-kube-api-access-qcbvl\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.295760 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295738    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7l7\" (UniqueName: \"kubernetes.io/projected/a8e9ef4a-e99c-4243-bd91-fbe366802378-kube-api-access-rq7l7\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-conf-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295787    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-node-log\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295811    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fn55\" (UniqueName: \"kubernetes.io/projected/3f913015-45d9-4ed6-9eae-931aaaefb6fd-kube-api-access-5fn55\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295851    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-systemd\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295875    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgrf\" (UniqueName: \"kubernetes.io/projected/b257fe13-9d6c-4f51-824f-3777dc25bff8-kube-api-access-5zgrf\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295898    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-kubelet\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295923    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-host\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295946    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbe84e1-dd8f-4701-93af-2087896382b3-host\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295971    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-device-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.295994    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296018    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-cni-multus\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296043    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-etc-kubernetes\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.296064 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296067    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296104    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovn-node-metrics-cert\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296130    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-cnibin\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296177    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-host-slash\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296206    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-daemon-config\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296231    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296254    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-systemd\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296286    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296307    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2rzb\" (UniqueName: \"kubernetes.io/projected/8ab68d45-73d4-4b34-a74c-aedcab554e1b-kube-api-access-m2rzb\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-hostroot\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296348    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-device-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a50581ee-e188-4fad-8c5b-96896f8e15b4-kubelet-config\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296395    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-kubernetes\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296422    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysctl-conf\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296446    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296470    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.296704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296500    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-socket-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296523    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-systemd-units\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296550    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfbj\" (UniqueName: \"kubernetes.io/projected/00248c3f-bd71-4073-a6bb-de0a31df55f4-kube-api-access-2nfbj\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296577    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-registration-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-system-cni-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296628    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-log-socket\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296654    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-modprobe-d\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296679    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-multus-certs\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296704    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90b2d32e-fadd-4603-9036-138eefcb0411-hosts-file\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296736    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-conf-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296729    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdkh\" (UniqueName: \"kubernetes.io/projected/90b2d32e-fadd-4603-9036-138eefcb0411-kube-api-access-frdkh\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296789    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296900    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-multus-certs\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.296927    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297153    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-socket-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297198    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-cni-multus\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-etc-kubernetes\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.297578 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297267    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-iptables-alerter-script\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz8tv\" (UniqueName: \"kubernetes.io/projected/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-kube-api-access-zz8tv\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297349    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-kubelet\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-slash\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297414    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-var-lib-kubelet\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297440    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-tuned\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297465    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297492    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-daemon-config\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-sys-fs\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-sys-fs\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-cni-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297584    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-netns\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297604    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-host-slash\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297619    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-ovn\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297648    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-cni-bin\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297674    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a50581ee-e188-4fad-8c5b-96896f8e15b4-dbus\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297716    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42747b31-db23-4a26-85cd-ccd997d13d0c-agent-certs\") pod \"konnectivity-agent-c8x87\" (UID: \"42747b31-db23-4a26-85cd-ccd997d13d0c\") " pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:04.298410 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297742    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysctl-d\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297765    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-sys\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297793    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b2d32e-fadd-4603-9036-138eefcb0411-tmp-dir\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297823    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6kb\" (UniqueName: \"kubernetes.io/projected/d594aab7-c878-4797-b9a3-2e04a55a1586-kube-api-access-rt6kb\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297851    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-lib-modules\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297896    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-cnibin\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.297925    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-socket-dir-parent\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298040    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-cni-netd\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298068    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-run\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298099    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d594aab7-c878-4797-b9a3-2e04a55a1586-cni-binary-copy\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298130    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-run-netns\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298158    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovnkube-script-lib\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298185    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298185    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-iptables-alerter-script\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298319    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-registration-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298415    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-system-cni-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298423    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-kubelet\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.298996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298460    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-hostroot\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298483    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysconfig\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298512    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-cni-dir\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298515    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fdbe84e1-dd8f-4701-93af-2087896382b3-serviceca\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-netns\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298584    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-cni-bin\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298598    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a8e9ef4a-e99c-4243-bd91-fbe366802378-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298613    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298640    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-env-overrides\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298667    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab68d45-73d4-4b34-a74c-aedcab554e1b-tmp\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298672    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-multus-socket-dir-parent\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298699    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-k8s-cni-cncf-io\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298730    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-var-lib-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298757    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-etc-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.298941    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-run-k8s-cni-cncf-io\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.299002    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-cnibin\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300007 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.299055    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d594aab7-c878-4797-b9a3-2e04a55a1586-host-var-lib-cni-bin\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.300899 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.299531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d594aab7-c878-4797-b9a3-2e04a55a1586-cni-binary-copy\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.311724 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.311695    2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:54:04.312367 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.312328    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gvzmc"
Sep 17 15:54:04.315925 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.315902    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7l7\" (UniqueName: \"kubernetes.io/projected/a8e9ef4a-e99c-4243-bd91-fbe366802378-kube-api-access-rq7l7\") pod \"aws-ebs-csi-driver-node-tzt9c\" (UID: \"a8e9ef4a-e99c-4243-bd91-fbe366802378\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.316034 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.315921    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz8tv\" (UniqueName: \"kubernetes.io/projected/50a00b6e-f5c7-4f8a-88fe-8611d25ae671-kube-api-access-zz8tv\") pod \"iptables-alerter-8bqtq\" (UID: \"50a00b6e-f5c7-4f8a-88fe-8611d25ae671\") " pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.320243 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.320197    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6kb\" (UniqueName: \"kubernetes.io/projected/d594aab7-c878-4797-b9a3-2e04a55a1586-kube-api-access-rt6kb\") pod \"multus-fqvct\" (UID: \"d594aab7-c878-4797-b9a3-2e04a55a1586\") " pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.326300 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.326280    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gvzmc"
Sep 17 15:54:04.327953 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.327929    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:04.328060 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.327957    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:04.328060 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.327971    2584 projected.go:196] Error preparing data for projected volume kube-api-access-mwh8g for pod openshift-network-diagnostics/network-check-target-h9dg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:04.328060 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.328048    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g podName:6fba78fa-721c-4986-ac7f-e7c994335e4b nodeName:}" failed. No retries permitted until 2026-09-17 15:54:04.828019314 +0000 UTC m=+3.054688603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mwh8g" (UniqueName: "kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g") pod "network-check-target-h9dg9" (UID: "6fba78fa-721c-4986-ac7f-e7c994335e4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:04.371495 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.371436    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal" event={"ID":"42f9cee7deb9c35203a248ac01b72173","Type":"ContainerStarted","Data":"7d4db4a6589f2b01982e1b68e3b87c17ccca288237f53d98f4fab0e239af6466"}
Sep 17 15:54:04.372675 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.372617    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal" event={"ID":"cddee824a28fdff15b30d083c82658d1","Type":"ContainerStarted","Data":"b4f1d4328d7f8c8d98ab8f4991714962855ecb9913d2e6a50412e42c7feeb283"}
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.399889    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-systemd-units\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.399939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfbj\" (UniqueName: \"kubernetes.io/projected/00248c3f-bd71-4073-a6bb-de0a31df55f4-kube-api-access-2nfbj\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400014    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-systemd-units\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-log-socket\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400205    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-modprobe-d\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400234    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90b2d32e-fadd-4603-9036-138eefcb0411-hosts-file\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400248    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-log-socket\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400395    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-modprobe-d\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400260    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdkh\" (UniqueName: \"kubernetes.io/projected/90b2d32e-fadd-4603-9036-138eefcb0411-kube-api-access-frdkh\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400450    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/90b2d32e-fadd-4603-9036-138eefcb0411-hosts-file\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400526    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-slash\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400553    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-var-lib-kubelet\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400589    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-tuned\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.400641 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400645    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-ovn\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-cni-bin\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400695    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a50581ee-e188-4fad-8c5b-96896f8e15b4-dbus\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400719    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42747b31-db23-4a26-85cd-ccd997d13d0c-agent-certs\") pod \"konnectivity-agent-c8x87\" (UID: \"42747b31-db23-4a26-85cd-ccd997d13d0c\") " pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400742    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysctl-d\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400767    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-sys\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400791    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b2d32e-fadd-4603-9036-138eefcb0411-tmp-dir\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-lib-modules\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400844    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-cni-netd\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-run\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400913    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-run-netns\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400929    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovnkube-script-lib\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400947    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400967    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysconfig\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.400990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fdbe84e1-dd8f-4701-93af-2087896382b3-serviceca\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401016    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401040    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-env-overrides\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401068    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab68d45-73d4-4b34-a74c-aedcab554e1b-tmp\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.401506 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401096    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-var-lib-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401114    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-etc-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401131    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovnkube-config\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401147    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42747b31-db23-4a26-85cd-ccd997d13d0c-konnectivity-ca\") pod \"konnectivity-agent-c8x87\" (UID: \"42747b31-db23-4a26-85cd-ccd997d13d0c\") " pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401163    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-system-cni-dir\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401189    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401219    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcbvl\" (UniqueName: \"kubernetes.io/projected/fdbe84e1-dd8f-4701-93af-2087896382b3-kube-api-access-qcbvl\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401248    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-node-log\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401275    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fn55\" (UniqueName: \"kubernetes.io/projected/3f913015-45d9-4ed6-9eae-931aaaefb6fd-kube-api-access-5fn55\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401300    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-systemd\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401334    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgrf\" (UniqueName: \"kubernetes.io/projected/b257fe13-9d6c-4f51-824f-3777dc25bff8-kube-api-access-5zgrf\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401360    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-kubelet\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401385    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-host\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401409    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbe84e1-dd8f-4701-93af-2087896382b3-host\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401469    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovn-node-metrics-cert\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-cnibin\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.402491 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401543    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-systemd\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401573    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401594    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2rzb\" (UniqueName: \"kubernetes.io/projected/8ab68d45-73d4-4b34-a74c-aedcab554e1b-kube-api-access-m2rzb\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401612    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a50581ee-e188-4fad-8c5b-96896f8e15b4-kubelet-config\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401627    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-kubernetes\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401643    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysctl-conf\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401660    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401677    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401906    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-host\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401929    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-var-lib-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401968    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-etc-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.401981    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-lib-modules\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.402030    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.402192    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs podName:b257fe13-9d6c-4f51-824f-3777dc25bff8 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:04.902165431 +0000 UTC m=+3.128834718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs") pod "network-metrics-daemon-wn4lc" (UID: "b257fe13-9d6c-4f51-824f-3777dc25bff8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402232    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-node-log\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402376    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-slash\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402417    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90b2d32e-fadd-4603-9036-138eefcb0411-tmp-dir\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.403260 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402430    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-var-lib-kubelet\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402458    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402518    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-run\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-run-ovn-kubernetes\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402604    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-run-netns\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402741    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovnkube-config\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402823    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-system-cni-dir\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402901    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42747b31-db23-4a26-85cd-ccd997d13d0c-konnectivity-ca\") pod \"konnectivity-agent-c8x87\" (UID: \"42747b31-db23-4a26-85cd-ccd997d13d0c\") " pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403045    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-systemd\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403260    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-openvswitch\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403307    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovnkube-script-lib\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403390    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysconfig\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403402    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403418    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-kubelet\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403438    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-cni-netd\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.402518    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdbe84e1-dd8f-4701-93af-2087896382b3-host\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403489    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a50581ee-e188-4fad-8c5b-96896f8e15b4-dbus\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.403543    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:04.404124 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403548    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysctl-d\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.403598    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret podName:a50581ee-e188-4fad-8c5b-96896f8e15b4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:04.903579554 +0000 UTC m=+3.130248841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret") pod "global-pull-secret-syncer-w8xbr" (UID: "a50581ee-e188-4fad-8c5b-96896f8e15b4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403680    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-cnibin\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403708    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-sys\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403732    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-systemd\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403772    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-run-ovn\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403782    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403835    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3f913015-45d9-4ed6-9eae-931aaaefb6fd-host-cni-bin\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403852    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a50581ee-e188-4fad-8c5b-96896f8e15b4-kubelet-config\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403881    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fdbe84e1-dd8f-4701-93af-2087896382b3-serviceca\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.403928    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-kubernetes\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.404000    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/00248c3f-bd71-4073-a6bb-de0a31df55f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.404043    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3f913015-45d9-4ed6-9eae-931aaaefb6fd-env-overrides\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.404052    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-sysctl-conf\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.405036 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.404073    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/00248c3f-bd71-4073-a6bb-de0a31df55f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.405837 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.405803    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3f913015-45d9-4ed6-9eae-931aaaefb6fd-ovn-node-metrics-cert\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.406360 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.406337    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42747b31-db23-4a26-85cd-ccd997d13d0c-agent-certs\") pod \"konnectivity-agent-c8x87\" (UID: \"42747b31-db23-4a26-85cd-ccd997d13d0c\") " pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:04.406657 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.406635    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ab68d45-73d4-4b34-a74c-aedcab554e1b-tmp\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.406827 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.406763    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ab68d45-73d4-4b34-a74c-aedcab554e1b-etc-tuned\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.412395 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.412338    2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:04.428463 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.428110    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2rzb\" (UniqueName: \"kubernetes.io/projected/8ab68d45-73d4-4b34-a74c-aedcab554e1b-kube-api-access-m2rzb\") pod \"tuned-t95wm\" (UID: \"8ab68d45-73d4-4b34-a74c-aedcab554e1b\") " pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.430732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.430707    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgrf\" (UniqueName: \"kubernetes.io/projected/b257fe13-9d6c-4f51-824f-3777dc25bff8-kube-api-access-5zgrf\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:04.430866 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.430767    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdkh\" (UniqueName: \"kubernetes.io/projected/90b2d32e-fadd-4603-9036-138eefcb0411-kube-api-access-frdkh\") pod \"node-resolver-n4hh7\" (UID: \"90b2d32e-fadd-4603-9036-138eefcb0411\") " pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.433691 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.433669    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfbj\" (UniqueName: \"kubernetes.io/projected/00248c3f-bd71-4073-a6bb-de0a31df55f4-kube-api-access-2nfbj\") pod \"multus-additional-cni-plugins-kxc2j\" (UID: \"00248c3f-bd71-4073-a6bb-de0a31df55f4\") " pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.436951 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.436780    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcbvl\" (UniqueName: \"kubernetes.io/projected/fdbe84e1-dd8f-4701-93af-2087896382b3-kube-api-access-qcbvl\") pod \"node-ca-g82l6\" (UID: \"fdbe84e1-dd8f-4701-93af-2087896382b3\") " pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.441423 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.438026    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fn55\" (UniqueName: \"kubernetes.io/projected/3f913015-45d9-4ed6-9eae-931aaaefb6fd-kube-api-access-5fn55\") pod \"ovnkube-node-szc22\" (UID: \"3f913015-45d9-4ed6-9eae-931aaaefb6fd\") " pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.494327 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.494299    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c"
Sep 17 15:54:04.506478 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.506456    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8bqtq"
Sep 17 15:54:04.518396 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.518366    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fqvct"
Sep 17 15:54:04.523399 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.523369    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:04.533652 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.533634    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n4hh7"
Sep 17 15:54:04.544864 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.544842    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:04.561989 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.561969    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t95wm"
Sep 17 15:54:04.571224 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.571207    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g82l6"
Sep 17 15:54:04.579028 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.579008    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kxc2j"
Sep 17 15:54:04.906923 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.906878    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:04.907118 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.906944    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:04.907118 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:04.906989    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:04.907118 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.907055    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:04.907286 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.907145    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:04.907286 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.907164    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs podName:b257fe13-9d6c-4f51-824f-3777dc25bff8 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:05.90713628 +0000 UTC m=+4.133805567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs") pod "network-metrics-daemon-wn4lc" (UID: "b257fe13-9d6c-4f51-824f-3777dc25bff8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:04.907286 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.907200    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:04.907286 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.907207    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret podName:a50581ee-e188-4fad-8c5b-96896f8e15b4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:05.907190285 +0000 UTC m=+4.133859571 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret") pod "global-pull-secret-syncer-w8xbr" (UID: "a50581ee-e188-4fad-8c5b-96896f8e15b4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:04.907286 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.907220    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:04.907286 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.907233    2584 projected.go:196] Error preparing data for projected volume kube-api-access-mwh8g for pod openshift-network-diagnostics/network-check-target-h9dg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:04.907286 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:04.907277    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g podName:6fba78fa-721c-4986-ac7f-e7c994335e4b nodeName:}" failed. No retries permitted until 2026-09-17 15:54:05.907264961 +0000 UTC m=+4.133934242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwh8g" (UniqueName: "kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g") pod "network-check-target-h9dg9" (UID: "6fba78fa-721c-4986-ac7f-e7c994335e4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:05.329434 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:05.329383    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:49:05 +0000 UTC" deadline="2028-04-03 14:36:33.774803287 +0000 UTC"
Sep 17 15:54:05.329434 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:05.329426    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13534h42m28.445382918s"
Sep 17 15:54:05.365798 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:05.365741    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:05.365966 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.365878    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:05.610999 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.610969    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e9ef4a_e99c_4243_bd91_fbe366802378.slice/crio-6cded4af26229d8b854a9e1c825ad0357cf3702cc4d4a58a3492d5a99b017d3f WatchSource:0}: Error finding container 6cded4af26229d8b854a9e1c825ad0357cf3702cc4d4a58a3492d5a99b017d3f: Status 404 returned error can't find the container with id 6cded4af26229d8b854a9e1c825ad0357cf3702cc4d4a58a3492d5a99b017d3f
Sep 17 15:54:05.611321 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.611291    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a00b6e_f5c7_4f8a_88fe_8611d25ae671.slice/crio-53f5225393d0b644b77732b7506d476dbbb8edd81b71c7d270829a4b69d5cc37 WatchSource:0}: Error finding container 53f5225393d0b644b77732b7506d476dbbb8edd81b71c7d270829a4b69d5cc37: Status 404 returned error can't find the container with id 53f5225393d0b644b77732b7506d476dbbb8edd81b71c7d270829a4b69d5cc37
Sep 17 15:54:05.616532 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.616512    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd594aab7_c878_4797_b9a3_2e04a55a1586.slice/crio-f290c82769d024209e696e29e2bc287b81202a289b732c73f51d41d418194298 WatchSource:0}: Error finding container f290c82769d024209e696e29e2bc287b81202a289b732c73f51d41d418194298: Status 404 returned error can't find the container with id f290c82769d024209e696e29e2bc287b81202a289b732c73f51d41d418194298
Sep 17 15:54:05.618035 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.618015    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbe84e1_dd8f_4701_93af_2087896382b3.slice/crio-87c7d624de3191db29aca738179fd91b925b2f7cb43ef16ed5358f841d658fdd WatchSource:0}: Error finding container 87c7d624de3191db29aca738179fd91b925b2f7cb43ef16ed5358f841d658fdd: Status 404 returned error can't find the container with id 87c7d624de3191db29aca738179fd91b925b2f7cb43ef16ed5358f841d658fdd
Sep 17 15:54:05.620727 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.620251    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00248c3f_bd71_4073_a6bb_de0a31df55f4.slice/crio-c51bb574f9f71a0beef474db78f2fbbd38739b8f7de3e0ee6e5e30d000f8597e WatchSource:0}: Error finding container c51bb574f9f71a0beef474db78f2fbbd38739b8f7de3e0ee6e5e30d000f8597e: Status 404 returned error can't find the container with id c51bb574f9f71a0beef474db78f2fbbd38739b8f7de3e0ee6e5e30d000f8597e
Sep 17 15:54:05.620727 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.620602    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f913015_45d9_4ed6_9eae_931aaaefb6fd.slice/crio-45ea9d3a4bee7e57fc28bfc94db31615b72439bf1d792d5c90fde7fd69fb669f WatchSource:0}: Error finding container 45ea9d3a4bee7e57fc28bfc94db31615b72439bf1d792d5c90fde7fd69fb669f: Status 404 returned error can't find the container with id 45ea9d3a4bee7e57fc28bfc94db31615b72439bf1d792d5c90fde7fd69fb669f
Sep 17 15:54:05.621992 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.621969    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b2d32e_fadd_4603_9036_138eefcb0411.slice/crio-b69e2db3b62ed00e46e677b9f9487ac782f786aeb869228d17ca28d70e651588 WatchSource:0}: Error finding container b69e2db3b62ed00e46e677b9f9487ac782f786aeb869228d17ca28d70e651588: Status 404 returned error can't find the container with id b69e2db3b62ed00e46e677b9f9487ac782f786aeb869228d17ca28d70e651588
Sep 17 15:54:05.623823 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.623776    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42747b31_db23_4a26_85cd_ccd997d13d0c.slice/crio-636019c574468501178d24b1b6f83c720356f9f528504e7762e45aff6dd9b57f WatchSource:0}: Error finding container 636019c574468501178d24b1b6f83c720356f9f528504e7762e45aff6dd9b57f: Status 404 returned error can't find the container with id 636019c574468501178d24b1b6f83c720356f9f528504e7762e45aff6dd9b57f
Sep 17 15:54:05.624790 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:05.624743    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab68d45_73d4_4b34_a74c_aedcab554e1b.slice/crio-12aa1ef4943c1b21c2435986554c2eff79ad63e17fe216a442fe35403f9b2409 WatchSource:0}: Error finding container 12aa1ef4943c1b21c2435986554c2eff79ad63e17fe216a442fe35403f9b2409: Status 404 returned error can't find the container with id 12aa1ef4943c1b21c2435986554c2eff79ad63e17fe216a442fe35403f9b2409
Sep 17 15:54:05.914741 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:05.914542    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:05.914848 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:05.914779    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:05.914848 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.914784    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:05.914848 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.914825    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:05.914848 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.914839    2584 projected.go:196] Error preparing data for projected volume kube-api-access-mwh8g for pod openshift-network-diagnostics/network-check-target-h9dg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:05.915089 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:05.914818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:05.915089 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.914901    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g podName:6fba78fa-721c-4986-ac7f-e7c994335e4b nodeName:}" failed. No retries permitted until 2026-09-17 15:54:07.914881209 +0000 UTC m=+6.141550480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwh8g" (UniqueName: "kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g") pod "network-check-target-h9dg9" (UID: "6fba78fa-721c-4986-ac7f-e7c994335e4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:05.915089 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.914970    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:05.915089 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.915026    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret podName:a50581ee-e188-4fad-8c5b-96896f8e15b4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:07.9150127 +0000 UTC m=+6.141681968 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret") pod "global-pull-secret-syncer-w8xbr" (UID: "a50581ee-e188-4fad-8c5b-96896f8e15b4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:05.915089 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.915043    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:05.915265 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:05.915099    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs podName:b257fe13-9d6c-4f51-824f-3777dc25bff8 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:07.915082747 +0000 UTC m=+6.141752018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs") pod "network-metrics-daemon-wn4lc" (UID: "b257fe13-9d6c-4f51-824f-3777dc25bff8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:05.962580 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:05.962551    2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:54:06.330100 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.330020    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:49:05 +0000 UTC" deadline="2028-06-20 23:40:54.476293688 +0000 UTC"
Sep 17 15:54:06.330100 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.330095    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15415h46m48.146203492s"
Sep 17 15:54:06.366934 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.366900    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:06.367120 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:06.367051    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:06.367588 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.367566    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:06.367708 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:06.367687    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:06.383067 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.382971    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t95wm" event={"ID":"8ab68d45-73d4-4b34-a74c-aedcab554e1b","Type":"ContainerStarted","Data":"12aa1ef4943c1b21c2435986554c2eff79ad63e17fe216a442fe35403f9b2409"}
Sep 17 15:54:06.386231 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.386195    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c8x87" event={"ID":"42747b31-db23-4a26-85cd-ccd997d13d0c","Type":"ContainerStarted","Data":"636019c574468501178d24b1b6f83c720356f9f528504e7762e45aff6dd9b57f"}
Sep 17 15:54:06.390934 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.390897    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n4hh7" event={"ID":"90b2d32e-fadd-4603-9036-138eefcb0411","Type":"ContainerStarted","Data":"b69e2db3b62ed00e46e677b9f9487ac782f786aeb869228d17ca28d70e651588"}
Sep 17 15:54:06.393431 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.393399    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerStarted","Data":"c51bb574f9f71a0beef474db78f2fbbd38739b8f7de3e0ee6e5e30d000f8597e"}
Sep 17 15:54:06.398900 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.398870    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c" event={"ID":"a8e9ef4a-e99c-4243-bd91-fbe366802378","Type":"ContainerStarted","Data":"6cded4af26229d8b854a9e1c825ad0357cf3702cc4d4a58a3492d5a99b017d3f"}
Sep 17 15:54:06.402587 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.402558    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal" event={"ID":"42f9cee7deb9c35203a248ac01b72173","Type":"ContainerStarted","Data":"de12af8cc20ba32223b16661f7111ab8d52aa3ef696e7f437db4f66a7ba5d3cd"}
Sep 17 15:54:06.405762 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.405734    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g82l6" event={"ID":"fdbe84e1-dd8f-4701-93af-2087896382b3","Type":"ContainerStarted","Data":"87c7d624de3191db29aca738179fd91b925b2f7cb43ef16ed5358f841d658fdd"}
Sep 17 15:54:06.414168 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.414118    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"45ea9d3a4bee7e57fc28bfc94db31615b72439bf1d792d5c90fde7fd69fb669f"}
Sep 17 15:54:06.423385 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.423321    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fqvct" event={"ID":"d594aab7-c878-4797-b9a3-2e04a55a1586","Type":"ContainerStarted","Data":"f290c82769d024209e696e29e2bc287b81202a289b732c73f51d41d418194298"}
Sep 17 15:54:06.441980 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.441945    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8bqtq" event={"ID":"50a00b6e-f5c7-4f8a-88fe-8611d25ae671","Type":"ContainerStarted","Data":"53f5225393d0b644b77732b7506d476dbbb8edd81b71c7d270829a4b69d5cc37"}
Sep 17 15:54:06.452876 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.452593    2584 generic.go:356] "Generic (PLEG): container finished" podID="cddee824a28fdff15b30d083c82658d1" containerID="ef4977a20845b1c08b91fc4974f0236393e19f433dd3b1286b4ce0e87c0ba49c" exitCode=0
Sep 17 15:54:06.452876 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.452646    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal" event={"ID":"cddee824a28fdff15b30d083c82658d1","Type":"ContainerDied","Data":"ef4977a20845b1c08b91fc4974f0236393e19f433dd3b1286b4ce0e87c0ba49c"}
Sep 17 15:54:06.478413 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:06.478344    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-128.ec2.internal" podStartSLOduration=2.4783155470000002 podStartE2EDuration="2.478315547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:06.425366599 +0000 UTC m=+4.652035892" watchObservedRunningTime="2026-09-17 15:54:06.478315547 +0000 UTC m=+4.704984840"
Sep 17 15:54:07.365509 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:07.365474    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:07.366162 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.365627    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:07.464011 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:07.463929    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal" event={"ID":"cddee824a28fdff15b30d083c82658d1","Type":"ContainerStarted","Data":"2e1144e73c6e286f49c9335fb09a93e2151df6bd7341af4b2de357a26bd29fc7"}
Sep 17 15:54:07.933018 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:07.932980    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:07.933263 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:07.933044    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:07.933263 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:07.933105    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:07.933406 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.933268    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:07.933406 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.933291    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:07.933406 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.933305    2584 projected.go:196] Error preparing data for projected volume kube-api-access-mwh8g for pod openshift-network-diagnostics/network-check-target-h9dg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:07.933406 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.933376    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g podName:6fba78fa-721c-4986-ac7f-e7c994335e4b nodeName:}" failed. No retries permitted until 2026-09-17 15:54:11.933349613 +0000 UTC m=+10.160018886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwh8g" (UniqueName: "kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g") pod "network-check-target-h9dg9" (UID: "6fba78fa-721c-4986-ac7f-e7c994335e4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:07.933620 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.933461    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:07.933620 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.933501    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs podName:b257fe13-9d6c-4f51-824f-3777dc25bff8 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:11.933487069 +0000 UTC m=+10.160156336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs") pod "network-metrics-daemon-wn4lc" (UID: "b257fe13-9d6c-4f51-824f-3777dc25bff8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:07.933620 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.933557    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:07.933620 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:07.933592    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret podName:a50581ee-e188-4fad-8c5b-96896f8e15b4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:11.933580647 +0000 UTC m=+10.160249916 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret") pod "global-pull-secret-syncer-w8xbr" (UID: "a50581ee-e188-4fad-8c5b-96896f8e15b4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:08.366721 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:08.366674    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:08.367232 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:08.366844    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:08.367306 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:08.367269    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:08.367409 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:08.367385    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:09.366436 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:09.365917    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:09.366436 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:09.366056    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:10.366009 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:10.365976    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:10.366507 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:10.366113    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:10.366698 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:10.366671    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:10.366955 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:10.366828    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:11.365842 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:11.365633    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:11.365842 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.365782    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:11.968948 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:11.968897    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:11.969590 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:11.968981    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:11.969590 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:11.969027    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:11.969590 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.969171    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:11.969590 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.969246    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret podName:a50581ee-e188-4fad-8c5b-96896f8e15b4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:19.9692258 +0000 UTC m=+18.195895069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret") pod "global-pull-secret-syncer-w8xbr" (UID: "a50581ee-e188-4fad-8c5b-96896f8e15b4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:11.969789 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.969696    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:11.969789 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.969720    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:11.969789 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.969735    2584 projected.go:196] Error preparing data for projected volume kube-api-access-mwh8g for pod openshift-network-diagnostics/network-check-target-h9dg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:11.969789 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.969785    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g podName:6fba78fa-721c-4986-ac7f-e7c994335e4b nodeName:}" failed. No retries permitted until 2026-09-17 15:54:19.969769039 +0000 UTC m=+18.196438323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwh8g" (UniqueName: "kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g") pod "network-check-target-h9dg9" (UID: "6fba78fa-721c-4986-ac7f-e7c994335e4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:11.969974 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.969867    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:11.969974 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:11.969904    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs podName:b257fe13-9d6c-4f51-824f-3777dc25bff8 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:19.969891301 +0000 UTC m=+18.196560571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs") pod "network-metrics-daemon-wn4lc" (UID: "b257fe13-9d6c-4f51-824f-3777dc25bff8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:12.366454 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:12.365898    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:12.366454 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:12.366057    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:12.367304 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:12.367150    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:12.367304 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:12.367259    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:13.366262 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:13.366224    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:13.366769 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:13.366356    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:14.366576 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:14.366172    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:14.366576 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:14.366327    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:14.367082 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:14.366803    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:14.367082 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:14.366887    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:15.366513 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:15.366347    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:15.366513 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:15.366477    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:16.365999 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:16.365955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:16.366193 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:16.365955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:16.366193 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:16.366108    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:16.366193 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:16.366169    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:17.366202 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:17.366125    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:17.366586 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:17.366232    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:18.365897 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:18.365865    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:18.366429 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:18.366366    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:18.366808 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:18.366465    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:18.367517 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:18.367395    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:19.366449 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:19.366415    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:19.366945 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:19.366560    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:20.028943 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:20.028911    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:20.029138 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:20.028971    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:20.029138 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:20.029006    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:20.029138 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.029133    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:20.029296 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.029200    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs podName:b257fe13-9d6c-4f51-824f-3777dc25bff8 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:36.029179462 +0000 UTC m=+34.255848749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs") pod "network-metrics-daemon-wn4lc" (UID: "b257fe13-9d6c-4f51-824f-3777dc25bff8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:20.029493 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.029466    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:20.029493 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.029496    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:20.029672 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.029509    2584 projected.go:196] Error preparing data for projected volume kube-api-access-mwh8g for pod openshift-network-diagnostics/network-check-target-h9dg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:20.029672 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.029551    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g podName:6fba78fa-721c-4986-ac7f-e7c994335e4b nodeName:}" failed. No retries permitted until 2026-09-17 15:54:36.029538401 +0000 UTC m=+34.256207684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwh8g" (UniqueName: "kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g") pod "network-check-target-h9dg9" (UID: "6fba78fa-721c-4986-ac7f-e7c994335e4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:20.029672 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.029613    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:20.029672 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.029656    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret podName:a50581ee-e188-4fad-8c5b-96896f8e15b4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:36.029643927 +0000 UTC m=+34.256313198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret") pod "global-pull-secret-syncer-w8xbr" (UID: "a50581ee-e188-4fad-8c5b-96896f8e15b4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:20.366061 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:20.366024    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:20.366240 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:20.366024    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:20.366240 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.366137    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:20.366336 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:20.366241    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:21.365720 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:21.365683    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:21.366120 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:21.365804    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:22.366937 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:22.366897    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:22.367398 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:22.367019    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:22.367398 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:22.367080    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:22.367398 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:22.367179    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:23.365916 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:23.365884    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:23.366073 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:23.365978    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:24.366396 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.366102    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:24.366770 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.366102    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:24.366770 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:24.366507    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:24.366770 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:24.366535    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:24.505343 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.505301    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t95wm" event={"ID":"8ab68d45-73d4-4b34-a74c-aedcab554e1b","Type":"ContainerStarted","Data":"e1ba9dcde7466e9ce398c66ee85151d0e657f3b399cc79eec0336dc57c53faa5"}
Sep 17 15:54:24.506847 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.506815    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c8x87" event={"ID":"42747b31-db23-4a26-85cd-ccd997d13d0c","Type":"ContainerStarted","Data":"43d559008216c98053a95049da3a820fb25c286e2b96a2926fc1aba955536316"}
Sep 17 15:54:24.508245 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.508224    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n4hh7" event={"ID":"90b2d32e-fadd-4603-9036-138eefcb0411","Type":"ContainerStarted","Data":"678b01b18856c9f1295ab1a55d174baf8825fe044e5eadfe8e7d04ea1adf085f"}
Sep 17 15:54:24.509877 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.509854    2584 generic.go:356] "Generic (PLEG): container finished" podID="00248c3f-bd71-4073-a6bb-de0a31df55f4" containerID="3589e03e31aa22f39fe3af05d8086498699704d40248eb69ebf1a3d740f2f7b7" exitCode=0
Sep 17 15:54:24.509973 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.509933    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerDied","Data":"3589e03e31aa22f39fe3af05d8086498699704d40248eb69ebf1a3d740f2f7b7"}
Sep 17 15:54:24.511526 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.511504    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c" event={"ID":"a8e9ef4a-e99c-4243-bd91-fbe366802378","Type":"ContainerStarted","Data":"e4336a2207dc563a2b574f7e34b2eea3dab2c7464a9503f5510aae85043a82e0"}
Sep 17 15:54:24.513157 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.513134    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g82l6" event={"ID":"fdbe84e1-dd8f-4701-93af-2087896382b3","Type":"ContainerStarted","Data":"961c565d39c539f688633aaff5f853ed7bbcfa6ca549bdc1384c13b031208812"}
Sep 17 15:54:24.516298 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.516273    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"e726c1c9725decff998753b2bbf6e930df02628fe234cbce85114ee0dc550c3b"}
Sep 17 15:54:24.516384 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.516347    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"541e1c972c65ca0e3e81e75fb7613afec02d920ac4e83359a1c0fbc8e367603a"}
Sep 17 15:54:24.516448 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.516395    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"614d4ab4470577f8e72ea90c7a32ab5ded1ac0c672834c0e0667daff360aee14"}
Sep 17 15:54:24.516448 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.516442    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"1c68bbc745a2f2ed6f0fe721fc87fa1c8712744c234546419f4497a9c26e6fc7"}
Sep 17 15:54:24.516546 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.516488    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"f191d2da4b2e03e5b2666b1fec2dd7e0af6a0c7c62273b562204f6ed04435187"}
Sep 17 15:54:24.518085 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.518066    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fqvct" event={"ID":"d594aab7-c878-4797-b9a3-2e04a55a1586","Type":"ContainerStarted","Data":"81e914a0cf7f5b8ddcb2edc666f9ff60496b5f2bb35e526717e2e96cdbd02393"}
Sep 17 15:54:24.545191 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.545151    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t95wm" podStartSLOduration=3.1885327119999998 podStartE2EDuration="21.545137311s" totalImagesPullingTime="18.356604599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:24.545008656 +0000 UTC m=+22.771677963" watchObservedRunningTime="2026-09-17 15:54:24.545137311 +0000 UTC m=+22.771806597"
Sep 17 15:54:24.545541 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.545516    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-128.ec2.internal" podStartSLOduration=20.545508247 podStartE2EDuration="20.545508247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:07.487834111 +0000 UTC m=+5.714503428" watchObservedRunningTime="2026-09-17 15:54:24.545508247 +0000 UTC m=+22.772177542"
Sep 17 15:54:24.572547 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.572494    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fqvct" podStartSLOduration=3.16817793 podStartE2EDuration="21.572475933s" totalImagesPullingTime="18.404298003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:24.571633148 +0000 UTC m=+22.798302454" watchObservedRunningTime="2026-09-17 15:54:24.572475933 +0000 UTC m=+22.799145222"
Sep 17 15:54:24.593003 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.592954    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n4hh7" podStartSLOduration=3.249528697 podStartE2EDuration="21.592934106s" totalImagesPullingTime="18.343405409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:24.591249966 +0000 UTC m=+22.817919273" watchObservedRunningTime="2026-09-17 15:54:24.592934106 +0000 UTC m=+22.819603395"
Sep 17 15:54:24.644584 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.644534    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c8x87" podStartSLOduration=3.303284652 podStartE2EDuration="21.644512961s" totalImagesPullingTime="18.341228309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:24.615353056 +0000 UTC m=+22.842022564" watchObservedRunningTime="2026-09-17 15:54:24.644512961 +0000 UTC m=+22.871182253"
Sep 17 15:54:24.686330 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:24.686272    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g82l6" podStartSLOduration=3.336701572 podStartE2EDuration="21.686251216s" totalImagesPullingTime="18.349549644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:24.644668799 +0000 UTC m=+22.871338073" watchObservedRunningTime="2026-09-17 15:54:24.686251216 +0000 UTC m=+22.912920508"
Sep 17 15:54:25.337246 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:25.337219    2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:54:25.366012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:25.365983    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:25.366162 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:25.366107    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:25.523270 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:25.523174    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"ddb735c97a7b7b6c3e650ec24f85f7457b7ca6eeb61654889f0e44f5130816aa"}
Sep 17 15:54:25.524720 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:25.524689    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8bqtq" event={"ID":"50a00b6e-f5c7-4f8a-88fe-8611d25ae671","Type":"ContainerStarted","Data":"940a572dbe025959e0a7ec65d725aa943864df9e43d111f3315c09d3317b4111"}
Sep 17 15:54:25.526465 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:25.526432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c" event={"ID":"a8e9ef4a-e99c-4243-bd91-fbe366802378","Type":"ContainerStarted","Data":"cecd7e9f0d20a8e3014e2f7b5cbf6d91ec73b53cd981ea7b68d93e7b6bd8cb4d"}
Sep 17 15:54:26.256427 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.256324    2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:54:25.337242267Z","UUID":"4f3bbd5a-0417-4a94-ba2a-178e6dc32866","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:54:26.259440 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.259408    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:54:26.259581 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.259448    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:54:26.366411 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.366386    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:26.366411 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.366398    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:26.366568 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:26.366502    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:26.366682 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:26.366649    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:26.531910 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.531820    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c" event={"ID":"a8e9ef4a-e99c-4243-bd91-fbe366802378","Type":"ContainerStarted","Data":"d7ff6c444d9f9593017e15047a469186fd2f0eb0b94def0c578ed340a20fb543"}
Sep 17 15:54:26.561171 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.561122    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8bqtq" podStartSLOduration=5.205498402 podStartE2EDuration="23.561100898s" totalImagesPullingTime="18.355602496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:25.546178973 +0000 UTC m=+23.772848261" watchObservedRunningTime="2026-09-17 15:54:26.561100898 +0000 UTC m=+24.787770189"
Sep 17 15:54:26.982985 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.982948    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:26.983553 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:26.983535    2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:27.016997 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:27.016931    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzt9c" podStartSLOduration=3.641724685 podStartE2EDuration="24.016911787s" totalImagesPullingTime="20.375187102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:26.56083612 +0000 UTC m=+24.787505433" watchObservedRunningTime="2026-09-17 15:54:27.016911787 +0000 UTC m=+25.243581089"
Sep 17 15:54:27.366419 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:27.366236    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:27.366594 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:27.366505    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:27.538479 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:27.538435    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"9451549f8ed4828e9a9d0ad2a720ad1676d456d1441a86e796fdb15dc73f1970"}
Sep 17 15:54:27.538946 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:27.538836    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:27.539265 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:27.539245    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c8x87"
Sep 17 15:54:28.366586 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:28.366555    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:28.366830 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:28.366560    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:28.366830 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:28.366671    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:28.366830 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:28.366768    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:29.366483 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:29.366262    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:29.366932 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:29.366524    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:29.543901 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:29.543864    2584 generic.go:356] "Generic (PLEG): container finished" podID="00248c3f-bd71-4073-a6bb-de0a31df55f4" containerID="0105886c9418bafb4ae2735498d67c0775064ff1efd1d8fa3e737c16b7b95e8f" exitCode=0
Sep 17 15:54:29.544049 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:29.543916    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerDied","Data":"0105886c9418bafb4ae2735498d67c0775064ff1efd1d8fa3e737c16b7b95e8f"}
Sep 17 15:54:29.548058 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:29.548037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" event={"ID":"3f913015-45d9-4ed6-9eae-931aaaefb6fd","Type":"ContainerStarted","Data":"1aa2dbeaff08be10c2afaeb15c72e359708753d92dca2de7ca701e3c110fca9f"}
Sep 17 15:54:29.655432 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:29.655328    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" podStartSLOduration=8.254476426 podStartE2EDuration="26.655308393s" totalImagesPullingTime="18.400831967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:29.654326076 +0000 UTC m=+27.880995367" watchObservedRunningTime="2026-09-17 15:54:29.655308393 +0000 UTC m=+27.881977681"
Sep 17 15:54:30.366184 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:30.366148    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:30.366411 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:30.366296    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:30.366411 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:30.366331    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:30.366526 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:30.366410    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:30.550431 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:30.550406    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:54:30.551048 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:30.550908    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:30.551048 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:30.550994    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:30.566203 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:30.566177    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:30.566334 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:30.566302    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:31.129479 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.129447    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8xbr"]
Sep 17 15:54:31.129705 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.129567    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:31.129705 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:31.129681    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:31.132908 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.132879    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wn4lc"]
Sep 17 15:54:31.133055 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.132991    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:31.133128 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:31.133105    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:31.169084 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.169051    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h9dg9"]
Sep 17 15:54:31.169238 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.169160    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:31.169280 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:31.169238    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:31.555003 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.554912    2584 generic.go:356] "Generic (PLEG): container finished" podID="00248c3f-bd71-4073-a6bb-de0a31df55f4" containerID="fc27bc8218ff1d4c5ac5c6e802ec3ba8567354aca0368c1bd943f7e65c1925f6" exitCode=0
Sep 17 15:54:31.555003 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.554992    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerDied","Data":"fc27bc8218ff1d4c5ac5c6e802ec3ba8567354aca0368c1bd943f7e65c1925f6"}
Sep 17 15:54:31.555766 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:31.555068    2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:54:32.221114 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:32.221076    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:54:32.367236 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:32.367205    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:32.367390 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:32.367303    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:32.560255 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:32.560227    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerStarted","Data":"b802358e56f62f12e21ede4dd83c02577bdfa349e03d4a7c7b23c13916979a62"}
Sep 17 15:54:33.365707 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:33.365676    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:33.365918 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:33.365773    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:33.365918 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:33.365831    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:33.366012 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:33.365926    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:33.565083 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:33.564906    2584 generic.go:356] "Generic (PLEG): container finished" podID="00248c3f-bd71-4073-a6bb-de0a31df55f4" containerID="b802358e56f62f12e21ede4dd83c02577bdfa349e03d4a7c7b23c13916979a62" exitCode=0
Sep 17 15:54:33.565083 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:33.564979    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerDied","Data":"b802358e56f62f12e21ede4dd83c02577bdfa349e03d4a7c7b23c13916979a62"}
Sep 17 15:54:33.582357 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:33.582327    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-szc22" podUID="3f913015-45d9-4ed6-9eae-931aaaefb6fd" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 17 15:54:34.365704 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:34.365667    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:34.365884 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:34.365837    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:35.365517 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:35.365482    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:35.365517 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:35.365518    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:35.366027 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:35.365605    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h9dg9" podUID="6fba78fa-721c-4986-ac7f-e7c994335e4b"
Sep 17 15:54:35.366027 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:35.365712    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w8xbr" podUID="a50581ee-e188-4fad-8c5b-96896f8e15b4"
Sep 17 15:54:36.047012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.046974    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:36.047211 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.047025    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:36.047211 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.047058    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:36.047211 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.047153    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:36.047211 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.047173    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:36.047211 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.047176    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:54:36.047445 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.047226    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:54:36.047445 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.047237    2584 projected.go:196] Error preparing data for projected volume kube-api-access-mwh8g for pod openshift-network-diagnostics/network-check-target-h9dg9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:36.047445 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.047226    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret podName:a50581ee-e188-4fad-8c5b-96896f8e15b4 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:08.047207559 +0000 UTC m=+66.273876829 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret") pod "global-pull-secret-syncer-w8xbr" (UID: "a50581ee-e188-4fad-8c5b-96896f8e15b4") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:54:36.047445 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.047270    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs podName:b257fe13-9d6c-4f51-824f-3777dc25bff8 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:08.047259299 +0000 UTC m=+66.273928566 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs") pod "network-metrics-daemon-wn4lc" (UID: "b257fe13-9d6c-4f51-824f-3777dc25bff8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:54:36.047445 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.047280    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g podName:6fba78fa-721c-4986-ac7f-e7c994335e4b nodeName:}" failed. No retries permitted until 2026-09-17 15:55:08.047274066 +0000 UTC m=+66.273943332 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mwh8g" (UniqueName: "kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g") pod "network-check-target-h9dg9" (UID: "6fba78fa-721c-4986-ac7f-e7c994335e4b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:54:36.366391 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.366358    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:36.366899 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.366483    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wn4lc" podUID="b257fe13-9d6c-4f51-824f-3777dc25bff8"
Sep 17 15:54:36.570421 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.570383    2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-128.ec2.internal" event="NodeReady"
Sep 17 15:54:36.570614 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.570550    2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:54:36.641116 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.641031    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"]
Sep 17 15:54:36.646583 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.646556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.660678 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.660643    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 15:54:36.666289 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.666255    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 15:54:36.666462 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.666373    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 15:54:36.666462 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.666265    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 15:54:36.666684 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.666664    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 15:54:36.667348 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.667322    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 15:54:36.667348 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.667342    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 15:54:36.668313 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.668288    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f46588648-5d9sl"]
Sep 17 15:54:36.672266 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.672240    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"]
Sep 17 15:54:36.672396 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.672364    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.676115 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.676091    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"]
Sep 17 15:54:36.676234 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.676214    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.679919 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.679894    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"
Sep 17 15:54:36.681943 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.681911    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:54:36.683041 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.683019    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:54:36.686006 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.685984    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 17 15:54:36.693808 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.687960    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 15:54:36.693808 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.688375    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7jqdh\""
Sep 17 15:54:36.693808 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.688754    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:54:36.693808 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.691321    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"]
Sep 17 15:54:36.693808 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.691374    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"]
Sep 17 15:54:36.698436 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.698407    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:54:36.708383 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.708357    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ptcg8"]
Sep 17 15:54:36.712425 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.712399    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f46588648-5d9sl"]
Sep 17 15:54:36.712551 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.712516    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.723975 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.723943    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:54:36.724143 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.724000    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nhj85\""
Sep 17 15:54:36.724143 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.724102    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:54:36.726706 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.726684    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mwd2m"]
Sep 17 15:54:36.730222 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.730203    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mwd2m"
Sep 17 15:54:36.735394 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.735369    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:54:36.735533 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.735435    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:54:36.735533 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.735459    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-djqn9\""
Sep 17 15:54:36.735640 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.735585    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:54:36.742417 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.742395    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ptcg8"]
Sep 17 15:54:36.743195 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.743179    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"]
Sep 17 15:54:36.751747 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.751723    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.751879 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.751763    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nhq\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-kube-api-access-h9nhq\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.751879 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.751822    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.751879 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.751849    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5033fc00-d840-40f0-b06d-d5b983d36144-tmp\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.751879 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.751865    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5033fc00-d840-40f0-b06d-d5b983d36144-klusterlet-config\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.752051 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.751896    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsk99\" (UniqueName: \"kubernetes.io/projected/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-kube-api-access-tsk99\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.752051 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.751929    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-bound-sa-token\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.752051 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752004    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqc2\" (UniqueName: \"kubernetes.io/projected/199ef66d-8fb5-4744-a8d5-c28dc3acdf65-kube-api-access-skqc2\") pod \"managed-serviceaccount-addon-agent-7977488784-cftq9\" (UID: \"199ef66d-8fb5-4744-a8d5-c28dc3acdf65\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"
Sep 17 15:54:36.752051 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752029    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-certificates\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.752226 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752087    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-trusted-ca\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.752226 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752127    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-installation-pull-secrets\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.752226 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752161    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/199ef66d-8fb5-4744-a8d5-c28dc3acdf65-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7977488784-cftq9\" (UID: \"199ef66d-8fb5-4744-a8d5-c28dc3acdf65\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"
Sep 17 15:54:36.752330 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752265    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-ca\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.752330 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752295    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-hub\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.752397 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752325    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzp5\" (UniqueName: \"kubernetes.io/projected/5033fc00-d840-40f0-b06d-d5b983d36144-kube-api-access-rzzp5\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.752397 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752356    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.752397 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-image-registry-private-configuration\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.752537 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752433    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.752537 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.752469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8f1d2f9-071d-41d5-b89b-21f756d328c4-ca-trust-extracted\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.760369 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.760347    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mwd2m"]
Sep 17 15:54:36.853866 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.853831    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nhq\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-kube-api-access-h9nhq\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.854047 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.853889    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.854047 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.853914    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5033fc00-d840-40f0-b06d-d5b983d36144-tmp\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.854047 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.853938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5033fc00-d840-40f0-b06d-d5b983d36144-klusterlet-config\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.854047 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.853965    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.854047 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.853988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dbdb6a-2110-41c9-b537-ba98feba5474-tmp-dir\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.854047 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsk99\" (UniqueName: \"kubernetes.io/projected/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-kube-api-access-tsk99\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.854372 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.854054    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:36.854372 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.854078    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f46588648-5d9sl: secret "image-registry-tls" not found
Sep 17 15:54:36.854372 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.854148    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls podName:a8f1d2f9-071d-41d5-b89b-21f756d328c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:37.354128325 +0000 UTC m=+35.580797611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls") pod "image-registry-f46588648-5d9sl" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4") : secret "image-registry-tls" not found
Sep 17 15:54:36.854372 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854199    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-bound-sa-token\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.854372 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854245    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17dbdb6a-2110-41c9-b537-ba98feba5474-config-volume\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.854372 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854278    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqc2\" (UniqueName: \"kubernetes.io/projected/199ef66d-8fb5-4744-a8d5-c28dc3acdf65-kube-api-access-skqc2\") pod \"managed-serviceaccount-addon-agent-7977488784-cftq9\" (UID: \"199ef66d-8fb5-4744-a8d5-c28dc3acdf65\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"
Sep 17 15:54:36.854372 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854316    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-certificates\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.854372 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854340    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-trusted-ca\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854442    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-installation-pull-secrets\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854443    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5033fc00-d840-40f0-b06d-d5b983d36144-tmp\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txv58\" (UniqueName: \"kubernetes.io/projected/17dbdb6a-2110-41c9-b537-ba98feba5474-kube-api-access-txv58\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/199ef66d-8fb5-4744-a8d5-c28dc3acdf65-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7977488784-cftq9\" (UID: \"199ef66d-8fb5-4744-a8d5-c28dc3acdf65\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854531    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-ca\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-hub\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854564    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzp5\" (UniqueName: \"kubernetes.io/projected/5033fc00-d840-40f0-b06d-d5b983d36144-kube-api-access-rzzp5\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854581    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-image-registry-private-configuration\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854640    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8f1d2f9-071d-41d5-b89b-21f756d328c4-ca-trust-extracted\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854683    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854709    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff418b7a-d28f-45b6-a986-6855478a78c3-cert\") pod \"ingress-canary-mwd2m\" (UID: \"ff418b7a-d28f-45b6-a986-6855478a78c3\") " pod="openshift-ingress-canary/ingress-canary-mwd2m"
Sep 17 15:54:36.854732 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854733    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvrxh\" (UniqueName: \"kubernetes.io/projected/ff418b7a-d28f-45b6-a986-6855478a78c3-kube-api-access-qvrxh\") pod \"ingress-canary-mwd2m\" (UID: \"ff418b7a-d28f-45b6-a986-6855478a78c3\") " pod="openshift-ingress-canary/ingress-canary-mwd2m"
Sep 17 15:54:36.855365 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.854857    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-certificates\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.855365 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.855300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8f1d2f9-071d-41d5-b89b-21f756d328c4-ca-trust-extracted\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.855474 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.855456    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.855755 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.855653    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-trusted-ca\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.859447 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.859389    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-image-registry-private-configuration\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.859568 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.859448    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-ca\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.860084 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.860059    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5033fc00-d840-40f0-b06d-d5b983d36144-klusterlet-config\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.860209 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.860191    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-installation-pull-secrets\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.860291 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.860275    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.860426 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.860406    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/199ef66d-8fb5-4744-a8d5-c28dc3acdf65-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7977488784-cftq9\" (UID: \"199ef66d-8fb5-4744-a8d5-c28dc3acdf65\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"
Sep 17 15:54:36.860465 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.860447    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-hub\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.862050 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.862014    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.868420 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.868397    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nhq\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-kube-api-access-h9nhq\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.870094 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.870069    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsk99\" (UniqueName: \"kubernetes.io/projected/b77f7dc1-d27d-4582-9339-9e2ad7979d9e-kube-api-access-tsk99\") pod \"cluster-proxy-proxy-agent-9d984cc76-4dzfs\" (UID: \"b77f7dc1-d27d-4582-9339-9e2ad7979d9e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.871798 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.871762    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-bound-sa-token\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:36.874543 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.874524    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqc2\" (UniqueName: \"kubernetes.io/projected/199ef66d-8fb5-4744-a8d5-c28dc3acdf65-kube-api-access-skqc2\") pod \"managed-serviceaccount-addon-agent-7977488784-cftq9\" (UID: \"199ef66d-8fb5-4744-a8d5-c28dc3acdf65\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"
Sep 17 15:54:36.890419 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.890391    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzp5\" (UniqueName: \"kubernetes.io/projected/5033fc00-d840-40f0-b06d-d5b983d36144-kube-api-access-rzzp5\") pod \"klusterlet-addon-workmgr-5866d9d588-2qsp4\" (UID: \"5033fc00-d840-40f0-b06d-d5b983d36144\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:36.955671 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.955588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17dbdb6a-2110-41c9-b537-ba98feba5474-config-volume\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.955671 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.955642    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txv58\" (UniqueName: \"kubernetes.io/projected/17dbdb6a-2110-41c9-b537-ba98feba5474-kube-api-access-txv58\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.955901 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.955800    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff418b7a-d28f-45b6-a986-6855478a78c3-cert\") pod \"ingress-canary-mwd2m\" (UID: \"ff418b7a-d28f-45b6-a986-6855478a78c3\") " pod="openshift-ingress-canary/ingress-canary-mwd2m"
Sep 17 15:54:36.955901 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.955830    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvrxh\" (UniqueName: \"kubernetes.io/projected/ff418b7a-d28f-45b6-a986-6855478a78c3-kube-api-access-qvrxh\") pod \"ingress-canary-mwd2m\" (UID: \"ff418b7a-d28f-45b6-a986-6855478a78c3\") " pod="openshift-ingress-canary/ingress-canary-mwd2m"
Sep 17 15:54:36.955901 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.955891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.956018 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.955916    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dbdb6a-2110-41c9-b537-ba98feba5474-tmp-dir\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.956257 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.956228    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17dbdb6a-2110-41c9-b537-ba98feba5474-tmp-dir\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.956336 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.956286    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/17dbdb6a-2110-41c9-b537-ba98feba5474-config-volume\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.956386 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.956341    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:36.956426 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:36.956396    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls podName:17dbdb6a-2110-41c9-b537-ba98feba5474 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:37.456380001 +0000 UTC m=+35.683049267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls") pod "dns-default-ptcg8" (UID: "17dbdb6a-2110-41c9-b537-ba98feba5474") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:36.958802 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.958758    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff418b7a-d28f-45b6-a986-6855478a78c3-cert\") pod \"ingress-canary-mwd2m\" (UID: \"ff418b7a-d28f-45b6-a986-6855478a78c3\") " pod="openshift-ingress-canary/ingress-canary-mwd2m"
Sep 17 15:54:36.968126 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.968100    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:54:36.978753 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.978727    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvrxh\" (UniqueName: \"kubernetes.io/projected/ff418b7a-d28f-45b6-a986-6855478a78c3-kube-api-access-qvrxh\") pod \"ingress-canary-mwd2m\" (UID: \"ff418b7a-d28f-45b6-a986-6855478a78c3\") " pod="openshift-ingress-canary/ingress-canary-mwd2m"
Sep 17 15:54:36.982808 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.982771    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txv58\" (UniqueName: \"kubernetes.io/projected/17dbdb6a-2110-41c9-b537-ba98feba5474-kube-api-access-txv58\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:36.998218 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:36.998189    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:37.004821 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.004801    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"
Sep 17 15:54:37.039285 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.039250    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mwd2m"
Sep 17 15:54:37.359264 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.359222    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:37.359420 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:37.359380    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:37.359420 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:37.359400    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f46588648-5d9sl: secret "image-registry-tls" not found
Sep 17 15:54:37.359492 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:37.359452    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls podName:a8f1d2f9-071d-41d5-b89b-21f756d328c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:38.359437933 +0000 UTC m=+36.586107198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls") pod "image-registry-f46588648-5d9sl" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4") : secret "image-registry-tls" not found
Sep 17 15:54:37.366306 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.366281    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:54:37.366444 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.366428    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:54:37.372409 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.372382    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:54:37.378193 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.378172    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qr24f\""
Sep 17 15:54:37.379399 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.379364    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:54:37.379502 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.379451    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:54:37.460156 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.459883    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:37.460769 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:37.460017    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:37.460769 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:37.460304    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls podName:17dbdb6a-2110-41c9-b537-ba98feba5474 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:38.46027615 +0000 UTC m=+36.686945418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls") pod "dns-default-ptcg8" (UID: "17dbdb6a-2110-41c9-b537-ba98feba5474") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:37.577885 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.577854    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerStarted","Data":"8fc7fb94e9d139eac2b1ce3aa1088f4f39f4f3a9db7a77f947d0fd5baf9218ce"}
Sep 17 15:54:37.690720 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.690693    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9"]
Sep 17 15:54:37.696737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.696710    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mwd2m"]
Sep 17 15:54:37.703207 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.703180    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"]
Sep 17 15:54:37.703891 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:37.703869    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"]
Sep 17 15:54:37.706098 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:37.706074    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod199ef66d_8fb5_4744_a8d5_c28dc3acdf65.slice/crio-5f41f9b30fa81a69e2d578d7c5e164621653acaf6796a602fc51e4db94360a81 WatchSource:0}: Error finding container 5f41f9b30fa81a69e2d578d7c5e164621653acaf6796a602fc51e4db94360a81: Status 404 returned error can't find the container with id 5f41f9b30fa81a69e2d578d7c5e164621653acaf6796a602fc51e4db94360a81
Sep 17 15:54:37.706813 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:37.706724    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb77f7dc1_d27d_4582_9339_9e2ad7979d9e.slice/crio-4e1a733b8eaae7b684ac0f634c47ff3e9b3d262ddb91e6070821c60d360675fd WatchSource:0}: Error finding container 4e1a733b8eaae7b684ac0f634c47ff3e9b3d262ddb91e6070821c60d360675fd: Status 404 returned error can't find the container with id 4e1a733b8eaae7b684ac0f634c47ff3e9b3d262ddb91e6070821c60d360675fd
Sep 17 15:54:37.709012 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:37.708961    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5033fc00_d840_40f0_b06d_d5b983d36144.slice/crio-9b83d1b0551ce91cf1585206c95860da12cd46eb5c62566199861eb1d09102aa WatchSource:0}: Error finding container 9b83d1b0551ce91cf1585206c95860da12cd46eb5c62566199861eb1d09102aa: Status 404 returned error can't find the container with id 9b83d1b0551ce91cf1585206c95860da12cd46eb5c62566199861eb1d09102aa
Sep 17 15:54:37.709188 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:54:37.709170    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff418b7a_d28f_45b6_a986_6855478a78c3.slice/crio-31833e78ebdfc6844c6dfe956783a1af1579feaae4a21473b71fd5f9b4d52894 WatchSource:0}: Error finding container 31833e78ebdfc6844c6dfe956783a1af1579feaae4a21473b71fd5f9b4d52894: Status 404 returned error can't find the container with id 31833e78ebdfc6844c6dfe956783a1af1579feaae4a21473b71fd5f9b4d52894
Sep 17 15:54:38.368561 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.366908    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:54:38.368561 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.367976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:38.368561 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:38.368178    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:38.368561 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:38.368193    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f46588648-5d9sl: secret "image-registry-tls" not found
Sep 17 15:54:38.368561 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:38.368250    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls podName:a8f1d2f9-071d-41d5-b89b-21f756d328c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:40.368231363 +0000 UTC m=+38.594900645 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls") pod "image-registry-f46588648-5d9sl" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4") : secret "image-registry-tls" not found
Sep 17 15:54:38.371921 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.371672    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:54:38.371921 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.371706    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v7v5\""
Sep 17 15:54:38.469300 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.469261    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:38.469884 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:38.469861    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:38.470007 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:38.469936    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls podName:17dbdb6a-2110-41c9-b537-ba98feba5474 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:40.469915794 +0000 UTC m=+38.696585063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls") pod "dns-default-ptcg8" (UID: "17dbdb6a-2110-41c9-b537-ba98feba5474") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:38.585184 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.585045    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mwd2m" event={"ID":"ff418b7a-d28f-45b6-a986-6855478a78c3","Type":"ContainerStarted","Data":"31833e78ebdfc6844c6dfe956783a1af1579feaae4a21473b71fd5f9b4d52894"}
Sep 17 15:54:38.593496 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.593300    2584 generic.go:356] "Generic (PLEG): container finished" podID="00248c3f-bd71-4073-a6bb-de0a31df55f4" containerID="8fc7fb94e9d139eac2b1ce3aa1088f4f39f4f3a9db7a77f947d0fd5baf9218ce" exitCode=0
Sep 17 15:54:38.593496 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.593386    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerDied","Data":"8fc7fb94e9d139eac2b1ce3aa1088f4f39f4f3a9db7a77f947d0fd5baf9218ce"}
Sep 17 15:54:38.596239 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.596170    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" event={"ID":"b77f7dc1-d27d-4582-9339-9e2ad7979d9e","Type":"ContainerStarted","Data":"4e1a733b8eaae7b684ac0f634c47ff3e9b3d262ddb91e6070821c60d360675fd"}
Sep 17 15:54:38.598984 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.598930    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9" event={"ID":"199ef66d-8fb5-4744-a8d5-c28dc3acdf65","Type":"ContainerStarted","Data":"5f41f9b30fa81a69e2d578d7c5e164621653acaf6796a602fc51e4db94360a81"}
Sep 17 15:54:38.601208 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:38.601159    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4" event={"ID":"5033fc00-d840-40f0-b06d-d5b983d36144","Type":"ContainerStarted","Data":"9b83d1b0551ce91cf1585206c95860da12cd46eb5c62566199861eb1d09102aa"}
Sep 17 15:54:39.612418 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:39.612296    2584 generic.go:356] "Generic (PLEG): container finished" podID="00248c3f-bd71-4073-a6bb-de0a31df55f4" containerID="026091e13ff623ce044e378e4ac18543c06e97be3171f1be640ed9579af00af9" exitCode=0
Sep 17 15:54:39.612418 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:39.612361    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerDied","Data":"026091e13ff623ce044e378e4ac18543c06e97be3171f1be640ed9579af00af9"}
Sep 17 15:54:40.386631 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:40.386369    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:40.386872 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:40.386538    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:40.386872 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:40.386716    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f46588648-5d9sl: secret "image-registry-tls" not found
Sep 17 15:54:40.386872 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:40.386803    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls podName:a8f1d2f9-071d-41d5-b89b-21f756d328c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:44.386767773 +0000 UTC m=+42.613437048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls") pod "image-registry-f46588648-5d9sl" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4") : secret "image-registry-tls" not found
Sep 17 15:54:40.488050 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:40.488004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:40.488662 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:40.488235    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:40.488662 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:40.488307    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls podName:17dbdb6a-2110-41c9-b537-ba98feba5474 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:44.488287148 +0000 UTC m=+42.714956438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls") pod "dns-default-ptcg8" (UID: "17dbdb6a-2110-41c9-b537-ba98feba5474") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:40.620948 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:40.620910    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" event={"ID":"00248c3f-bd71-4073-a6bb-de0a31df55f4","Type":"ContainerStarted","Data":"e64e133ff16ebb16fc1190cd0c64bc77de071d58c933e98bfd0710937db04eb9"}
Sep 17 15:54:40.657376 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:40.657271    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kxc2j" podStartSLOduration=9.026832327 podStartE2EDuration="37.657250927s" totalImagesPullingTime="28.6304186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:40.656522081 +0000 UTC m=+38.883191372" watchObservedRunningTime="2026-09-17 15:54:40.657250927 +0000 UTC m=+38.883920216"
Sep 17 15:54:41.003530 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:41.003422    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:41.153034 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:41.152998    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:42.584005 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:42.583969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:42.678451 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:42.678416    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:44.420011 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:44.419976    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:44.420454 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:44.420167    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:44.420454 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:44.420189    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f46588648-5d9sl: secret "image-registry-tls" not found
Sep 17 15:54:44.420454 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:44.420258    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls podName:a8f1d2f9-071d-41d5-b89b-21f756d328c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:52.420238542 +0000 UTC m=+50.646907832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls") pod "image-registry-f46588648-5d9sl" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4") : secret "image-registry-tls" not found
Sep 17 15:54:44.520946 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:44.520907    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:44.521101 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:44.521061    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:44.521140 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:44.521126    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls podName:17dbdb6a-2110-41c9-b537-ba98feba5474 nodeName:}" failed. No retries permitted until 2026-09-17 15:54:52.521111917 +0000 UTC m=+50.747781183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls") pod "dns-default-ptcg8" (UID: "17dbdb6a-2110-41c9-b537-ba98feba5474") : secret "dns-default-metrics-tls" not found
Sep 17 15:54:44.652451 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:44.652411    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:44.846854 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:44.846761    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:45.035041 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:45.035008    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:45.164989 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:45.164904    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:45.294538 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:45.294500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:45.435048 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:45.434965    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:45.630835 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:45.630800    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:45.738418 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:45.738329    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:45.832305 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:45.832267    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:46.637133 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.637102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" event={"ID":"b77f7dc1-d27d-4582-9339-9e2ad7979d9e","Type":"ContainerStarted","Data":"d313c07be04852739e0ff964f889427a440f4e782f192cd9e13eb6d07a2548b4"}
Sep 17 15:54:46.638593 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.638564    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9" event={"ID":"199ef66d-8fb5-4744-a8d5-c28dc3acdf65","Type":"ContainerStarted","Data":"a892001a97ae118805cd064896a2ae6464bd6a93d888eaa2834db56f58c90aa5"}
Sep 17 15:54:46.640051 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.640028    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4" event={"ID":"5033fc00-d840-40f0-b06d-d5b983d36144","Type":"ContainerStarted","Data":"08110a05d212aa2cdcc6a4e4993443273c96799124dfcbeffb589063cc9794be"}
Sep 17 15:54:46.640178 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.640080    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:46.641653 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.641629    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mwd2m" event={"ID":"ff418b7a-d28f-45b6-a986-6855478a78c3","Type":"ContainerStarted","Data":"84984b275309aff31732d6dcab2e644cdb6dc0a1f832ccb8dfd7d108c0e1eb46"}
Sep 17 15:54:46.642006 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.641986    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:54:46.667576 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.667524    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9" podStartSLOduration=7.036372123 podStartE2EDuration="15.667505163s" totalImagesPullingTime="8.63113304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:46.667313881 +0000 UTC m=+44.893983170" watchObservedRunningTime="2026-09-17 15:54:46.667505163 +0000 UTC m=+44.894174455"
Sep 17 15:54:46.751478 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.751422    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4" podStartSLOduration=6.121987193 podStartE2EDuration="14.751403222s" totalImagesPullingTime="8.629416029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:46.751128424 +0000 UTC m=+44.977797713" watchObservedRunningTime="2026-09-17 15:54:46.751403222 +0000 UTC m=+44.978072561"
Sep 17 15:54:46.751889 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:46.751856    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mwd2m" podStartSLOduration=2.123317824 podStartE2EDuration="10.751845906s" totalImagesPullingTime="8.628528082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:46.707906614 +0000 UTC m=+44.934575928" watchObservedRunningTime="2026-09-17 15:54:46.751845906 +0000 UTC m=+44.978515197"
Sep 17 15:54:47.647072 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:47.647035    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" event={"ID":"b77f7dc1-d27d-4582-9339-9e2ad7979d9e","Type":"ContainerStarted","Data":"969431298b9522794b35ba1ea7bfbd367ae184588c2d8244f5b2239665009c26"}
Sep 17 15:54:47.647072 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:47.647078    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" event={"ID":"b77f7dc1-d27d-4582-9339-9e2ad7979d9e","Type":"ContainerStarted","Data":"845c8306aa450c1dcdc0f16e9312a58d11245a75ae8bfe8b5e19efcd0f3cdf9b"}
Sep 17 15:54:47.678015 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:47.677964    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" podStartSLOduration=7.032591722 podStartE2EDuration="15.677946412s" totalImagesPullingTime="8.64535469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:47.676123108 +0000 UTC m=+45.902792396" watchObservedRunningTime="2026-09-17 15:54:47.677946412 +0000 UTC m=+45.904615696"
Sep 17 15:54:48.545682 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:48.545645    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:48.634350 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:48.634319    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:50.222649 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:50.222612    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:50.306349 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:50.306289    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:54:52.481928 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:52.481887    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:54:52.482308 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:52.482060    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:54:52.482308 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:52.482080    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f46588648-5d9sl: secret "image-registry-tls" not found
Sep 17 15:54:52.482308 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:52.482147    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls podName:a8f1d2f9-071d-41d5-b89b-21f756d328c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:08.482131796 +0000 UTC m=+66.708801061 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls") pod "image-registry-f46588648-5d9sl" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4") : secret "image-registry-tls" not found
Sep 17 15:54:52.582609 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:54:52.582574    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:54:52.582759 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:52.582736    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:54:52.582848 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:54:52.582838    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls podName:17dbdb6a-2110-41c9-b537-ba98feba5474 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:08.582819586 +0000 UTC m=+66.809488856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls") pod "dns-default-ptcg8" (UID: "17dbdb6a-2110-41c9-b537-ba98feba5474") : secret "dns-default-metrics-tls" not found
Sep 17 15:55:03.574990 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:03.574962    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-szc22"
Sep 17 15:55:03.806313 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:03.806280    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:03.891683 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:03.891606    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:04.285302 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:04.285203    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:05.882829 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:05.882791    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:05.976746 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:05.976692    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:06.112086 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:06.112047    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:06.192176 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:06.192087    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:06.289008 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:06.288969    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:06.942916 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:06.942886    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:07.034429 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:07.034392    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:08.098579 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.098540    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:55:08.099057 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.098610    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:55:08.099057 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.098653    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:55:08.101679 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.101661    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:55:08.102132 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.102117    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:55:08.102449 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.102434    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:55:08.109645 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:08.109627    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:55:08.109715 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:08.109701    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs podName:b257fe13-9d6c-4f51-824f-3777dc25bff8 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:12.109683059 +0000 UTC m=+130.336352340 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs") pod "network-metrics-daemon-wn4lc" (UID: "b257fe13-9d6c-4f51-824f-3777dc25bff8") : secret "metrics-daemon-secret" not found
Sep 17 15:55:08.112225 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.112210    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:55:08.113055 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.113039    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a50581ee-e188-4fad-8c5b-96896f8e15b4-original-pull-secret\") pod \"global-pull-secret-syncer-w8xbr\" (UID: \"a50581ee-e188-4fad-8c5b-96896f8e15b4\") " pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:55:08.121908 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.121890    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh8g\" (UniqueName: \"kubernetes.io/projected/6fba78fa-721c-4986-ac7f-e7c994335e4b-kube-api-access-mwh8g\") pod \"network-check-target-h9dg9\" (UID: \"6fba78fa-721c-4986-ac7f-e7c994335e4b\") " pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:55:08.276567 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.276528    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w8xbr"
Sep 17 15:55:08.285346 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.285323    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qr24f\""
Sep 17 15:55:08.293131 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.293106    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:55:08.501711 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.501668    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:55:08.501896 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:08.501851    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:08.501896 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:08.501871    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f46588648-5d9sl: secret "image-registry-tls" not found
Sep 17 15:55:08.501966 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:08.501942    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls podName:a8f1d2f9-071d-41d5-b89b-21f756d328c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:40.501924997 +0000 UTC m=+98.728594262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls") pod "image-registry-f46588648-5d9sl" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4") : secret "image-registry-tls" not found
Sep 17 15:55:08.556845 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.556748    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w8xbr"]
Sep 17 15:55:08.560367 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:55:08.560344    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50581ee_e188_4fad_8c5b_96896f8e15b4.slice/crio-fef05865affaad653a77e50c65d61e3d5eb0eaa48cf84b412b75615a6313f953 WatchSource:0}: Error finding container fef05865affaad653a77e50c65d61e3d5eb0eaa48cf84b412b75615a6313f953: Status 404 returned error can't find the container with id fef05865affaad653a77e50c65d61e3d5eb0eaa48cf84b412b75615a6313f953
Sep 17 15:55:08.580079 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.580053    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h9dg9"]
Sep 17 15:55:08.583145 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:55:08.583119    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fba78fa_721c_4986_ac7f_e7c994335e4b.slice/crio-b5230566b717c1e942773a57c67fad13d0e0c5b405ead60edcfcd3004dd6029e WatchSource:0}: Error finding container b5230566b717c1e942773a57c67fad13d0e0c5b405ead60edcfcd3004dd6029e: Status 404 returned error can't find the container with id b5230566b717c1e942773a57c67fad13d0e0c5b405ead60edcfcd3004dd6029e
Sep 17 15:55:08.602205 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.602149    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:55:08.602296 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:08.602284    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:55:08.602345 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:08.602335    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls podName:17dbdb6a-2110-41c9-b537-ba98feba5474 nodeName:}" failed. No retries permitted until 2026-09-17 15:55:40.60232186 +0000 UTC m=+98.828991126 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls") pod "dns-default-ptcg8" (UID: "17dbdb6a-2110-41c9-b537-ba98feba5474") : secret "dns-default-metrics-tls" not found
Sep 17 15:55:08.707727 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.707697    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h9dg9" event={"ID":"6fba78fa-721c-4986-ac7f-e7c994335e4b","Type":"ContainerStarted","Data":"b5230566b717c1e942773a57c67fad13d0e0c5b405ead60edcfcd3004dd6029e"}
Sep 17 15:55:08.708770 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.708747    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8xbr" event={"ID":"a50581ee-e188-4fad-8c5b-96896f8e15b4","Type":"ContainerStarted","Data":"fef05865affaad653a77e50c65d61e3d5eb0eaa48cf84b412b75615a6313f953"}
Sep 17 15:55:08.866388 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:08.866301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:11.500990 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:11.500940    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:11.603488 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:11.603421    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:11.704104 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:11.704061    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:13.726879 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:13.726836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w8xbr" event={"ID":"a50581ee-e188-4fad-8c5b-96896f8e15b4","Type":"ContainerStarted","Data":"65f0bb02e86dbfb2f1bc090aa99d76b1416fda1ed943f1e1ea7288ee7c682358"}
Sep 17 15:55:13.728370 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:13.728346    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h9dg9" event={"ID":"6fba78fa-721c-4986-ac7f-e7c994335e4b","Type":"ContainerStarted","Data":"2c67bc53333e9d0b4a7ac94c7221b82b7981700e88158df1a2aa6b65b7f6ec51"}
Sep 17 15:55:13.728477 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:13.728468    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:55:13.751242 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:13.751195    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w8xbr" podStartSLOduration=66.167377092 podStartE2EDuration="1m10.751180219s" totalImagesPullingTime="4.583803127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:13.750995969 +0000 UTC m=+71.977665258" watchObservedRunningTime="2026-09-17 15:55:13.751180219 +0000 UTC m=+71.977849540"
Sep 17 15:55:13.779023 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:13.778974    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-h9dg9" podStartSLOduration=66.220877649 podStartE2EDuration="1m10.77895849s" totalImagesPullingTime="4.558080841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:55:13.778374986 +0000 UTC m=+72.005044301" watchObservedRunningTime="2026-09-17 15:55:13.77895849 +0000 UTC m=+72.005627778"
Sep 17 15:55:18.605841 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:18.605803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:20.369470 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:20.369425    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:20.986054 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:20.986014    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:21.089180 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:21.089143    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:21.194388 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:21.194356    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:23.056461 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:23.056426    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:23.140701 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:23.140664    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:33.461008 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:33.460880    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:33.527522 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:33.527486    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:40.543394 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:40.543356    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:55:40.543868 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:40.543512    2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:55:40.543868 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:40.543533    2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f46588648-5d9sl: secret "image-registry-tls" not found
Sep 17 15:55:40.543868 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:40.543592    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls podName:a8f1d2f9-071d-41d5-b89b-21f756d328c4 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:44.543577374 +0000 UTC m=+162.770246640 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls") pod "image-registry-f46588648-5d9sl" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4") : secret "image-registry-tls" not found
Sep 17 15:55:40.644171 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:40.644141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:55:40.644315 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:40.644292    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:55:40.644392 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:55:40.644382    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls podName:17dbdb6a-2110-41c9-b537-ba98feba5474 nodeName:}" failed. No retries permitted until 2026-09-17 15:56:44.644367894 +0000 UTC m=+162.871037160 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls") pod "dns-default-ptcg8" (UID: "17dbdb6a-2110-41c9-b537-ba98feba5474") : secret "dns-default-metrics-tls" not found
Sep 17 15:55:43.990737 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:43.990694    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:44.092877 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:44.092832    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:44.733930 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:44.733895    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h9dg9"
Sep 17 15:55:56.343961 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:56.343928    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:56.455111 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:56.455068    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:58.047346 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:58.047306    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:55:58.154633 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:55:58.154594    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:00.061745 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.061705    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:00.162696 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.162660    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:00.351468 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.351430    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jclm2"]
Sep 17 15:56:00.354426 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.354411    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.358046 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.358020    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:56:00.358290 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.358277    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:56:00.358575 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.358563    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:56:00.359139 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.359117    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:56:00.359139 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.359132    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hkffj\""
Sep 17 15:56:00.371250 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.371223    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:00.379733 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.379710    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jclm2"]
Sep 17 15:56:00.493055 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.493021    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2279r\" (UniqueName: \"kubernetes.io/projected/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-api-access-2279r\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.493233 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.493083    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.493233 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.493114    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.493233 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.493150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd10f39-f258-4875-93f7-d9d4a0b1192d-crio-socket\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.493447 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.493231    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fd10f39-f258-4875-93f7-d9d4a0b1192d-data-volume\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.507263 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.507229    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:00.594374 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.594337    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fd10f39-f258-4875-93f7-d9d4a0b1192d-data-volume\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.594374 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.594378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2279r\" (UniqueName: \"kubernetes.io/projected/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-api-access-2279r\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.594616 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.594415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.594616 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.594443    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.594616 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.594468    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd10f39-f258-4875-93f7-d9d4a0b1192d-crio-socket\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.594616 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:00.594537    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:56:00.594616 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.594555    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8fd10f39-f258-4875-93f7-d9d4a0b1192d-crio-socket\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.594616 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:00.594589    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:00.594616 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:00.594617    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:01.094601018 +0000 UTC m=+119.321270302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : configmap "kube-rbac-proxy" not found
Sep 17 15:56:00.594962 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:00.594669    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:01.094650993 +0000 UTC m=+119.321320279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:00.594962 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.594730    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8fd10f39-f258-4875-93f7-d9d4a0b1192d-data-volume\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.603417 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.603363    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2279r\" (UniqueName: \"kubernetes.io/projected/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-api-access-2279r\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:00.606305 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.606276    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:00.796575 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.796542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:00.920071 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:00.919984    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:01.098158 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:01.098109    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:01.098553 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:01.098173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:01.098553 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:01.098238    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:56:01.098553 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:01.098312    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:02.098296016 +0000 UTC m=+120.324965281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : configmap "kube-rbac-proxy" not found
Sep 17 15:56:01.098553 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:01.098319    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:01.098553 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:01.098361    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:02.098346633 +0000 UTC m=+120.325015899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:02.052622 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:02.052598    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n4hh7_90b2d32e-fadd-4603-9036-138eefcb0411/dns-node-resolver/0.log"
Sep 17 15:56:02.105773 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:02.105743    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:02.106118 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:02.105806    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:02.106118 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:02.105856    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:56:02.106118 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:02.105922    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:02.106118 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:02.105927    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:04.105912622 +0000 UTC m=+122.332581888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : configmap "kube-rbac-proxy" not found
Sep 17 15:56:02.106118 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:02.105972    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:04.105959267 +0000 UTC m=+122.332628533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:02.431088 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:02.431040    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:02.563296 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:02.563257    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:03.085588 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:03.085552    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:03.261592 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:03.261569    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g82l6_fdbe84e1-dd8f-4701-93af-2087896382b3/node-ca/0.log"
Sep 17 15:56:03.849812 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:03.849761    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:04.061149 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:04.061126    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mwd2m_ff418b7a-d28f-45b6-a986-6855478a78c3/serve-healthcheck-canary/0.log"
Sep 17 15:56:04.120317 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:04.120234    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:04.120317 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:04.120282    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:04.120473 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:04.120314    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:56:04.120473 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:04.120375    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:08.120360477 +0000 UTC m=+126.347029743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : configmap "kube-rbac-proxy" not found
Sep 17 15:56:04.120473 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:04.120397    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:04.120473 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:04.120438    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:08.120423819 +0000 UTC m=+126.347093085 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:04.296315 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:04.296285    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:04.449967 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:04.449878    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:04.569109 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:04.569075    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:04.691483 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:04.691452    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:04.798238 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:04.798154    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:05.105936 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:05.105901    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:05.522995 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:05.522915    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:06.089294 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:06.089261    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:06.518661 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:06.518574    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:06.706374 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:06.706343    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:07.127325 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:07.127291    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:07.933151 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:07.933122    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:08.059378 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.059349    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:08.149481 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.149438    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:08.149679 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.149495    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:08.149679 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:08.149566    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:56:08.149679 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:08.149631    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:16.14961546 +0000 UTC m=+134.376284727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : configmap "kube-rbac-proxy" not found
Sep 17 15:56:08.149679 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:08.149630    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:08.149679 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:08.149659    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls podName:8fd10f39-f258-4875-93f7-d9d4a0b1192d nodeName:}" failed. No retries permitted until 2026-09-17 15:56:16.149653205 +0000 UTC m=+134.376322471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jclm2" (UID: "8fd10f39-f258-4875-93f7-d9d4a0b1192d") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:56:08.205571 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.205487    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:08.309437 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.309403    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:08.484503 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.484417    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:08.695758 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.695721    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:08.784604 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.784516    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:08.902176 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:08.902149    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:09.894534 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:09.894475    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:10.089682 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:10.089649    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:10.306292 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:10.306216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:10.508271 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:10.508237    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:10.745853 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:10.745811    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:11.313406 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:11.313376    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:11.929631 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:11.929603    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:12.103723 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.103687    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:12.182360 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.182275    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:56:12.184629 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.184597    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b257fe13-9d6c-4f51-824f-3777dc25bff8-metrics-certs\") pod \"network-metrics-daemon-wn4lc\" (UID: \"b257fe13-9d6c-4f51-824f-3777dc25bff8\") " pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:56:12.285536 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.285510    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v7v5\""
Sep 17 15:56:12.292978 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.292955    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wn4lc"
Sep 17 15:56:12.329733 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.329675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:12.566960 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.566931    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wn4lc"]
Sep 17 15:56:12.572412 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:56:12.572378    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb257fe13_9d6c_4f51_824f_3777dc25bff8.slice/crio-6ba51cb2ea9392bcf4a96436e1eb5034bd03f0bc6299523644d44e5599a7bf88 WatchSource:0}: Error finding container 6ba51cb2ea9392bcf4a96436e1eb5034bd03f0bc6299523644d44e5599a7bf88: Status 404 returned error can't find the container with id 6ba51cb2ea9392bcf4a96436e1eb5034bd03f0bc6299523644d44e5599a7bf88
Sep 17 15:56:12.719845 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.719809    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:12.902179 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:12.902092    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wn4lc" event={"ID":"b257fe13-9d6c-4f51-824f-3777dc25bff8","Type":"ContainerStarted","Data":"6ba51cb2ea9392bcf4a96436e1eb5034bd03f0bc6299523644d44e5599a7bf88"}
Sep 17 15:56:13.589976 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:13.589923    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:13.726558 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:13.726512    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:13.907140 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:13.907102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wn4lc" event={"ID":"b257fe13-9d6c-4f51-824f-3777dc25bff8","Type":"ContainerStarted","Data":"493b132c9696d06d49a72fb134876c391739ad549d547ae370fb224f86c866cf"}
Sep 17 15:56:13.907304 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:13.907165    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wn4lc" event={"ID":"b257fe13-9d6c-4f51-824f-3777dc25bff8","Type":"ContainerStarted","Data":"f319c8c4881f1e81489c4bdd6226439269e005ea0d6d3f7113f8da93ff0bee2f"}
Sep 17 15:56:13.938018 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:13.937968    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wn4lc" podStartSLOduration=130.02363128 podStartE2EDuration="2m10.937952682s" totalImagesPullingTime="914.321402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:13.937297376 +0000 UTC m=+132.163966682" watchObservedRunningTime="2026-09-17 15:56:13.937952682 +0000 UTC m=+132.164621970"
Sep 17 15:56:14.245492 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:14.245402    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:14.366309 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:14.366269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:14.496291 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:14.496209    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:14.790911 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:14.790829    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:15.116929 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:15.116891    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:15.331284 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:15.331253    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:16.216219 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.216174    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:16.216640 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.216280    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:16.216853 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.216830    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8fd10f39-f258-4875-93f7-d9d4a0b1192d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:16.218546 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.218527    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8fd10f39-f258-4875-93f7-d9d4a0b1192d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jclm2\" (UID: \"8fd10f39-f258-4875-93f7-d9d4a0b1192d\") " pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:16.265379 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.265349    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hkffj\""
Sep 17 15:56:16.272691 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.272667    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jclm2"
Sep 17 15:56:16.554078 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.553980    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jclm2"]
Sep 17 15:56:16.557042 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:56:16.557012    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fd10f39_f258_4875_93f7_d9d4a0b1192d.slice/crio-5a9754d0fc8bae963c1bd29b6c1bbc067b336b9056f6f9dc0bda9aed520ddb0d WatchSource:0}: Error finding container 5a9754d0fc8bae963c1bd29b6c1bbc067b336b9056f6f9dc0bda9aed520ddb0d: Status 404 returned error can't find the container with id 5a9754d0fc8bae963c1bd29b6c1bbc067b336b9056f6f9dc0bda9aed520ddb0d
Sep 17 15:56:16.918576 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.918535    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jclm2" event={"ID":"8fd10f39-f258-4875-93f7-d9d4a0b1192d","Type":"ContainerStarted","Data":"0550614fb8f45e04380c0db68556dbb4e7e6cb829bac034f7b2be769fcb0a6ea"}
Sep 17 15:56:16.918763 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:16.918612    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jclm2" event={"ID":"8fd10f39-f258-4875-93f7-d9d4a0b1192d","Type":"ContainerStarted","Data":"5a9754d0fc8bae963c1bd29b6c1bbc067b336b9056f6f9dc0bda9aed520ddb0d"}
Sep 17 15:56:17.923885 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:17.923844    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jclm2" event={"ID":"8fd10f39-f258-4875-93f7-d9d4a0b1192d","Type":"ContainerStarted","Data":"aefce35b6f07bdcf9e07476b6fc909730f952f34ef3495519a772c9f01b28835"}
Sep 17 15:56:18.929128 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:18.929094    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jclm2" event={"ID":"8fd10f39-f258-4875-93f7-d9d4a0b1192d","Type":"ContainerStarted","Data":"525161d74aabea302b39caf375a2c7dd165a85d621ace8b278716c146723ac48"}
Sep 17 15:56:18.955961 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:18.955911    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jclm2" podStartSLOduration=17.273375203 podStartE2EDuration="18.955896085s" totalImagesPullingTime="1.682520882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:56:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:18.955285658 +0000 UTC m=+137.181954950" watchObservedRunningTime="2026-09-17 15:56:18.955896085 +0000 UTC m=+137.182565386"
Sep 17 15:56:21.249020 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:21.248982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:21.409254 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:21.409216    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:25.987604 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:25.987566    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:26.088055 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:26.088022    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:26.970571 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:26.970533    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" podUID="b77f7dc1-d27d-4582-9339-9e2ad7979d9e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:56:29.771022 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:29.770991    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:36.969501 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:36.969460    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" podUID="b77f7dc1-d27d-4582-9339-9e2ad7979d9e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:56:37.236053 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:37.235960    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:37.359942 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:37.359906    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:37.463502 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:37.463455    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:39.686152 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:39.686092    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-f46588648-5d9sl" podUID="a8f1d2f9-071d-41d5-b89b-21f756d328c4"
Sep 17 15:56:39.727766 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:56:39.727723    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ptcg8" podUID="17dbdb6a-2110-41c9-b537-ba98feba5474"
Sep 17 15:56:39.852895 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:39.852848    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:39.993981 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:39.993906    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:56:39.994115 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:39.993916    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:56:40.184644 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:40.184608    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:40.303950 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:40.303866    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:40.482353 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:40.482317    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:40.609973 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:40.609942    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:40.994849 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:40.994755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:41.748043 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:41.748002    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:41.830703 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:41.830661    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:41.913599 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:41.913575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:42.044367 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:42.044274    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:42.228079 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:42.228044    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:42.331986 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:42.331948    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:42.432052 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:42.432023    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:42.612592 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:42.612511    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:42.704402 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:42.704368    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:42.830644 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:42.830609    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:43.246215 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:43.246181    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:43.343656 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:43.343615    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:43.444356 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:43.444322    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:43.620059 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:43.620026    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:44.297708 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.297670    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:44.624416 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.624378    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:56:44.626631 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.626612    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"image-registry-f46588648-5d9sl\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") " pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:56:44.651900 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.651866    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:44.725548 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.725505    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:56:44.727821 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.727797    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/17dbdb6a-2110-41c9-b537-ba98feba5474-metrics-tls\") pod \"dns-default-ptcg8\" (UID: \"17dbdb6a-2110-41c9-b537-ba98feba5474\") " pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:56:44.798554 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.798523    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nhj85\""
Sep 17 15:56:44.798554 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.798523    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7jqdh\""
Sep 17 15:56:44.804910 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.804884    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:56:44.804977 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.804913    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:56:44.816089 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.816048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:44.911271 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:44.911191    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:45.006803 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.006749    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:45.028211 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.028185    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rr98r"]
Sep 17 15:56:45.033150 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.033132    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.035888 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.035868    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:56:45.035989 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.035868    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:56:45.036687 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.036671    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:56:45.037099 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.037079    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:56:45.037182 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.037134    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:56:45.037236 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.037180    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dc5q2\""
Sep 17 15:56:45.041250 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.041227    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:56:45.091289 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.091144    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ptcg8"]
Sep 17 15:56:45.093701 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:56:45.093680    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17dbdb6a_2110_41c9_b537_ba98feba5474.slice/crio-c4f35f900ddfa3a44358b38728b6815ab5a47d52775213df92cfb9e750f2c431 WatchSource:0}: Error finding container c4f35f900ddfa3a44358b38728b6815ab5a47d52775213df92cfb9e750f2c431: Status 404 returned error can't find the container with id c4f35f900ddfa3a44358b38728b6815ab5a47d52775213df92cfb9e750f2c431
Sep 17 15:56:45.097328 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.097305    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f46588648-5d9sl"]
Sep 17 15:56:45.098552 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:56:45.098530    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f1d2f9_071d_41d5_b89b_21f756d328c4.slice/crio-8e60fa55d9ef51a69d4ba2ecf97522ab4a22c3dc264bc7280cb44c3d57745b78 WatchSource:0}: Error finding container 8e60fa55d9ef51a69d4ba2ecf97522ab4a22c3dc264bc7280cb44c3d57745b78: Status 404 returned error can't find the container with id 8e60fa55d9ef51a69d4ba2ecf97522ab4a22c3dc264bc7280cb44c3d57745b78
Sep 17 15:56:45.111409 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.111381    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:45.128316 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128225    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.128316 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-sys\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.128316 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128303    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdpz\" (UniqueName: \"kubernetes.io/projected/f0971868-78f4-4808-a898-a9668294ed59-kube-api-access-gxdpz\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.128513 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128371    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-tls\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.128513 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128444    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-root\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.128513 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128470    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-wtmp\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.128513 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128497    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-textfile\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.128683 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128523    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0971868-78f4-4808-a898-a9668294ed59-metrics-client-ca\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.128683 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.128606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-accelerators-collector-config\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.229824 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.229725    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-root\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.229985 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.229823    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-root\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.229985 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.229859    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-wtmp\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.229985 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.229891    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-textfile\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.229985 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.229909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0971868-78f4-4808-a898-a9668294ed59-metrics-client-ca\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.229985 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.229944    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-accelerators-collector-config\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230210 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.229985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-wtmp\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230210 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.229990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230210 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.230079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-sys\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230210 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.230132    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdpz\" (UniqueName: \"kubernetes.io/projected/f0971868-78f4-4808-a898-a9668294ed59-kube-api-access-gxdpz\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230210 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.230173    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-tls\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230451 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.230298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-textfile\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230451 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.230355    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0971868-78f4-4808-a898-a9668294ed59-sys\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230583 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.230551    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-accelerators-collector-config\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.230635 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.230619    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0971868-78f4-4808-a898-a9668294ed59-metrics-client-ca\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.232453 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.232425    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.232573 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.232531    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0971868-78f4-4808-a898-a9668294ed59-node-exporter-tls\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.253363 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.253342    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdpz\" (UniqueName: \"kubernetes.io/projected/f0971868-78f4-4808-a898-a9668294ed59-kube-api-access-gxdpz\") pod \"node-exporter-rr98r\" (UID: \"f0971868-78f4-4808-a898-a9668294ed59\") " pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.260059 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.260032    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:45.342350 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.342315    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rr98r"
Sep 17 15:56:45.353505 ip-10-0-128-128 kubenswrapper[2584]: W0917 15:56:45.353480    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0971868_78f4_4808_a898_a9668294ed59.slice/crio-678782cdb5323c4466e42b4d351fb30c886f83921a56093adf57f8873ded0a95 WatchSource:0}: Error finding container 678782cdb5323c4466e42b4d351fb30c886f83921a56093adf57f8873ded0a95: Status 404 returned error can't find the container with id 678782cdb5323c4466e42b4d351fb30c886f83921a56093adf57f8873ded0a95
Sep 17 15:56:45.355294 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.355208    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:45.458219 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:45.458188    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:46.018193 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.018150    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rr98r" event={"ID":"f0971868-78f4-4808-a898-a9668294ed59","Type":"ContainerStarted","Data":"678782cdb5323c4466e42b4d351fb30c886f83921a56093adf57f8873ded0a95"}
Sep 17 15:56:46.020068 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.020037    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f46588648-5d9sl" event={"ID":"a8f1d2f9-071d-41d5-b89b-21f756d328c4","Type":"ContainerStarted","Data":"c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433"}
Sep 17 15:56:46.020182 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.020088    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f46588648-5d9sl" event={"ID":"a8f1d2f9-071d-41d5-b89b-21f756d328c4","Type":"ContainerStarted","Data":"8e60fa55d9ef51a69d4ba2ecf97522ab4a22c3dc264bc7280cb44c3d57745b78"}
Sep 17 15:56:46.020348 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.020331    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:56:46.021611 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.021586    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptcg8" event={"ID":"17dbdb6a-2110-41c9-b537-ba98feba5474","Type":"ContainerStarted","Data":"c4f35f900ddfa3a44358b38728b6815ab5a47d52775213df92cfb9e750f2c431"}
Sep 17 15:56:46.059436 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.059385    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f46588648-5d9sl" podStartSLOduration=162.059364081 podStartE2EDuration="2m42.059364081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:46.056988515 +0000 UTC m=+164.283657825" watchObservedRunningTime="2026-09-17 15:56:46.059364081 +0000 UTC m=+164.286033384"
Sep 17 15:56:46.061756 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.061687    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:46.143208 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.143174    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:46.640524 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.640472    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4" podUID="5033fc00-d840-40f0-b06d-d5b983d36144" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Sep 17 15:56:46.732369 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.732343    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:46.922510 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.922481    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:46.969443 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.969408    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" podUID="b77f7dc1-d27d-4582-9339-9e2ad7979d9e" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 17 15:56:46.969577 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.969474    2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs"
Sep 17 15:56:46.970077 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.970045    2584 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"969431298b9522794b35ba1ea7bfbd367ae184588c2d8244f5b2239665009c26"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 17 15:56:46.970126 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.970099    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" podUID="b77f7dc1-d27d-4582-9339-9e2ad7979d9e" containerName="service-proxy" containerID="cri-o://969431298b9522794b35ba1ea7bfbd367ae184588c2d8244f5b2239665009c26" gracePeriod=30
Sep 17 15:56:46.998878 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:46.998811    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4" podUID="5033fc00-d840-40f0-b06d-d5b983d36144" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/healthz\": dial tcp 10.132.0.10:8000: connect: connection refused"
Sep 17 15:56:47.005772 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.005749    2584 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9" podUID="199ef66d-8fb5-4744-a8d5-c28dc3acdf65" containerName="addon-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/healthz\": dial tcp 10.132.0.8:8000: connect: connection refused"
Sep 17 15:56:47.029216 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.029183    2584 generic.go:356] "Generic (PLEG): container finished" podID="199ef66d-8fb5-4744-a8d5-c28dc3acdf65" containerID="a892001a97ae118805cd064896a2ae6464bd6a93d888eaa2834db56f58c90aa5" exitCode=255
Sep 17 15:56:47.029364 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.029257    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9" event={"ID":"199ef66d-8fb5-4744-a8d5-c28dc3acdf65","Type":"ContainerDied","Data":"a892001a97ae118805cd064896a2ae6464bd6a93d888eaa2834db56f58c90aa5"}
Sep 17 15:56:47.029584 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.029561    2584 scope.go:117] "RemoveContainer" containerID="a892001a97ae118805cd064896a2ae6464bd6a93d888eaa2834db56f58c90aa5"
Sep 17 15:56:47.030899 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.030870    2584 generic.go:356] "Generic (PLEG): container finished" podID="f0971868-78f4-4808-a898-a9668294ed59" containerID="051708ca9047dfb88f187bece8dc9adbf5a071b5d968e15d928f0afc6940ee58" exitCode=0
Sep 17 15:56:47.030969 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.030911    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rr98r" event={"ID":"f0971868-78f4-4808-a898-a9668294ed59","Type":"ContainerDied","Data":"051708ca9047dfb88f187bece8dc9adbf5a071b5d968e15d928f0afc6940ee58"}
Sep 17 15:56:47.032565 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.032542    2584 generic.go:356] "Generic (PLEG): container finished" podID="5033fc00-d840-40f0-b06d-d5b983d36144" containerID="08110a05d212aa2cdcc6a4e4993443273c96799124dfcbeffb589063cc9794be" exitCode=1
Sep 17 15:56:47.032639 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.032614    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4" event={"ID":"5033fc00-d840-40f0-b06d-d5b983d36144","Type":"ContainerDied","Data":"08110a05d212aa2cdcc6a4e4993443273c96799124dfcbeffb589063cc9794be"}
Sep 17 15:56:47.032964 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.032945    2584 scope.go:117] "RemoveContainer" containerID="08110a05d212aa2cdcc6a4e4993443273c96799124dfcbeffb589063cc9794be"
Sep 17 15:56:47.034483 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.034462    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptcg8" event={"ID":"17dbdb6a-2110-41c9-b537-ba98feba5474","Type":"ContainerStarted","Data":"5ffc638f0671a9ba2524651b56f5d52ec2263aea06c1dc053b3a5fbaab4b249c"}
Sep 17 15:56:47.034586 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.034501    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ptcg8" event={"ID":"17dbdb6a-2110-41c9-b537-ba98feba5474","Type":"ContainerStarted","Data":"a57b6545875de0220f026e01bf0ccda63d5160d09aaf3e55059132aa5ef50aa7"}
Sep 17 15:56:47.094041 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:47.093988    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ptcg8" podStartSLOduration=129.453893687 podStartE2EDuration="2m11.093966667s" totalImagesPullingTime="1.64007298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:47.092718488 +0000 UTC m=+165.319387799" watchObservedRunningTime="2026-09-17 15:56:47.093966667 +0000 UTC m=+165.320635957"
Sep 17 15:56:48.040736 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.040638    2584 generic.go:356] "Generic (PLEG): container finished" podID="b77f7dc1-d27d-4582-9339-9e2ad7979d9e" containerID="969431298b9522794b35ba1ea7bfbd367ae184588c2d8244f5b2239665009c26" exitCode=2
Sep 17 15:56:48.040736 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.040711    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" event={"ID":"b77f7dc1-d27d-4582-9339-9e2ad7979d9e","Type":"ContainerDied","Data":"969431298b9522794b35ba1ea7bfbd367ae184588c2d8244f5b2239665009c26"}
Sep 17 15:56:48.041188 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.040755    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9d984cc76-4dzfs" event={"ID":"b77f7dc1-d27d-4582-9339-9e2ad7979d9e","Type":"ContainerStarted","Data":"419ca30c1db7044aeb2ed75297e83c4dc3e14b286e073e3beb97f6f7bb23e4ac"}
Sep 17 15:56:48.042419 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.042399    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7977488784-cftq9" event={"ID":"199ef66d-8fb5-4744-a8d5-c28dc3acdf65","Type":"ContainerStarted","Data":"040c6cdbc37d15f202e04a1f9661646b09d34763caa3367ae676d0854c23f712"}
Sep 17 15:56:48.044461 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.044443    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rr98r" event={"ID":"f0971868-78f4-4808-a898-a9668294ed59","Type":"ContainerStarted","Data":"71edc88c4feb5545679f01ca725a2d63b7ca2e9c37d7a316a1a9edb21b3f02a5"}
Sep 17 15:56:48.044536 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.044479    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rr98r" event={"ID":"f0971868-78f4-4808-a898-a9668294ed59","Type":"ContainerStarted","Data":"4da0de013f17491b1d8e2858891a0557b6d63bb1659030d67521aa0f1fd1f4a9"}
Sep 17 15:56:48.046230 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.046212    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4" event={"ID":"5033fc00-d840-40f0-b06d-d5b983d36144","Type":"ContainerStarted","Data":"2290a25d3d2fc4010eb20d6b39fba010b1c7121651336c640f2b787803ff56d6"}
Sep 17 15:56:48.046354 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.046343    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:56:48.046557 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.046543    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:56:48.047566 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.047551    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5866d9d588-2qsp4"
Sep 17 15:56:48.144759 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.144702    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rr98r" podStartSLOduration=1.738754194 podStartE2EDuration="3.14468471s" totalImagesPullingTime="1.33660986s" totalInitContainerRuntime="69.320656ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:48.141042105 +0000 UTC m=+166.367711394" watchObservedRunningTime="2026-09-17 15:56:48.14468471 +0000 UTC m=+166.371354040"
Sep 17 15:56:48.210493 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.210454    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:48.368447 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.368408    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:48.560745 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.560712    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:48.726088 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:48.726010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:49.696868 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:49.696828    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:49.796908 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:49.796868    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:50.228635 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:50.228600    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:51.485981 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:51.485941    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:55.847716 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:55.847634    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:55.950574 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:55.950531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:56.071932 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:56.071895    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:57.394280 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:57.394245    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:57.506654 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:57.506625    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:58.052006 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:58.051978    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ptcg8"
Sep 17 15:56:59.228254 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:59.228201    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:59.312022 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:59.311987    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:59.753054 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:59.753025    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f46588648-5d9sl"]
Sep 17 15:56:59.843636 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:59.843604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:56:59.943297 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:56:59.943259    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:00.044012 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:00.043921    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:02.426141 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:02.426110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:02.526949 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:02.526916    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:05.238936 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:05.238897    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:05.336618 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:05.336585    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:05.950254 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:05.950219    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:06.083108 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:06.083073    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:06.179631 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:06.179596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:06.593325 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:06.593290    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:06.688992 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:06.688959    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:07.230475 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:07.230436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:07.383751 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:07.383722    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:09.656455 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:09.656422    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:09.758864 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:09.758831    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:57:11.040805 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:11.040749    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:11.130118 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:11.130082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:17.019518 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:17.019483    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:17.089082 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:17.089042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:24.771489 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:24.771448    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f46588648-5d9sl" podUID="a8f1d2f9-071d-41d5-b89b-21f756d328c4" containerName="registry" containerID="cri-o://c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433" gracePeriod=30
Sep 17 15:57:25.002201 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.002178    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:57:25.010421 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.010401    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-installation-pull-secrets\") pod \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") "
Sep 17 15:57:25.010546 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.010429    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8f1d2f9-071d-41d5-b89b-21f756d328c4-ca-trust-extracted\") pod \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") "
Sep 17 15:57:25.010546 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.010451    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-bound-sa-token\") pod \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") "
Sep 17 15:57:25.010546 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.010471    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-certificates\") pod \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") "
Sep 17 15:57:25.010546 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.010492    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-image-registry-private-configuration\") pod \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") "
Sep 17 15:57:25.010546 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.010509    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9nhq\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-kube-api-access-h9nhq\") pod \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") "
Sep 17 15:57:25.010546 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.010535    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-trusted-ca\") pod \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") "
Sep 17 15:57:25.011074 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.011051    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a8f1d2f9-071d-41d5-b89b-21f756d328c4" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:25.011188 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.011048    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a8f1d2f9-071d-41d5-b89b-21f756d328c4" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:57:25.012844 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.012803    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a8f1d2f9-071d-41d5-b89b-21f756d328c4" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:25.012954 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.012875    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a8f1d2f9-071d-41d5-b89b-21f756d328c4" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:57:25.013053 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.013032    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a8f1d2f9-071d-41d5-b89b-21f756d328c4" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:25.013131 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.013108    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-kube-api-access-h9nhq" (OuterVolumeSpecName: "kube-api-access-h9nhq") pod "a8f1d2f9-071d-41d5-b89b-21f756d328c4" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4"). InnerVolumeSpecName "kube-api-access-h9nhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:25.018631 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.018609    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8f1d2f9-071d-41d5-b89b-21f756d328c4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a8f1d2f9-071d-41d5-b89b-21f756d328c4" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:57:25.110997 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.110964    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") pod \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\" (UID: \"a8f1d2f9-071d-41d5-b89b-21f756d328c4\") "
Sep 17 15:57:25.111154 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.111087    2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-installation-pull-secrets\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 15:57:25.111154 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.111100    2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8f1d2f9-071d-41d5-b89b-21f756d328c4-ca-trust-extracted\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 15:57:25.111154 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.111109    2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-bound-sa-token\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 15:57:25.111154 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.111118    2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-certificates\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 15:57:25.111154 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.111128    2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8f1d2f9-071d-41d5-b89b-21f756d328c4-image-registry-private-configuration\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 15:57:25.111154 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.111137    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9nhq\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-kube-api-access-h9nhq\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 15:57:25.111154 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.111145    2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8f1d2f9-071d-41d5-b89b-21f756d328c4-trusted-ca\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 15:57:25.112925 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.112897    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a8f1d2f9-071d-41d5-b89b-21f756d328c4" (UID: "a8f1d2f9-071d-41d5-b89b-21f756d328c4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:57:25.168635 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.168601    2584 generic.go:356] "Generic (PLEG): container finished" podID="a8f1d2f9-071d-41d5-b89b-21f756d328c4" containerID="c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433" exitCode=0
Sep 17 15:57:25.168773 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.168647    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f46588648-5d9sl"
Sep 17 15:57:25.168773 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.168683    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f46588648-5d9sl" event={"ID":"a8f1d2f9-071d-41d5-b89b-21f756d328c4","Type":"ContainerDied","Data":"c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433"}
Sep 17 15:57:25.168773 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.168737    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f46588648-5d9sl" event={"ID":"a8f1d2f9-071d-41d5-b89b-21f756d328c4","Type":"ContainerDied","Data":"8e60fa55d9ef51a69d4ba2ecf97522ab4a22c3dc264bc7280cb44c3d57745b78"}
Sep 17 15:57:25.168773 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.168757    2584 scope.go:117] "RemoveContainer" containerID="c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433"
Sep 17 15:57:25.176851 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.176834    2584 scope.go:117] "RemoveContainer" containerID="c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433"
Sep 17 15:57:25.177102 ip-10-0-128-128 kubenswrapper[2584]: E0917 15:57:25.177081    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433\": container with ID starting with c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433 not found: ID does not exist" containerID="c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433"
Sep 17 15:57:25.177148 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.177110    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433"} err="failed to get container status \"c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433\": rpc error: code = NotFound desc = could not find container \"c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433\": container with ID starting with c099006995594fb2cd273e6962ef96dae007791ee9be9c4381d1b72a84921433 not found: ID does not exist"
Sep 17 15:57:25.192610 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.192583    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f46588648-5d9sl"]
Sep 17 15:57:25.196366 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.196347    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f46588648-5d9sl"]
Sep 17 15:57:25.211404 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.211385    2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8f1d2f9-071d-41d5-b89b-21f756d328c4-registry-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 15:57:25.984249 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:25.984220    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:26.084691 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:26.084659    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:57:26.370684 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:57:26.370611    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f1d2f9-071d-41d5-b89b-21f756d328c4" path="/var/lib/kubelet/pods/a8f1d2f9-071d-41d5-b89b-21f756d328c4/volumes"
Sep 17 15:58:04.469413 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:04.469383    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:04.681180 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:04.681131    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:13.337753 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:13.337706    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:13.403256 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:13.403225    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:13.464776 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:13.464736    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:13.566227 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:13.566198    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:14.158762 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:14.158720    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:14.259616 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:14.259575    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:24.370615 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:24.370574    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:24.468259 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:24.468227    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:24.526243 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:24.526215    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:45.636494 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:45.636459    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:58:45.737955 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:58:45.737924    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:02.204060 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:02.204030    2584 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:59:19.749927 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:19.749882    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:19.817678 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:19.817640    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:31.668509 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:31.668472    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:31.739378 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:31.739335    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:38.484996 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:38.484957    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:38.547312 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:38.547282    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:40.206392 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:40.206358    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:40.981832 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:40.981726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:41.082527 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:41.082485    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:41.179800 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:41.179755    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:41.279904 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:41.279803    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:49.702677 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:49.702636    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:49.803926 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:49.803854    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 15:59:49.879423 ip-10-0-128-128 kubenswrapper[2584]: I0917 15:59:49.879393    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:07.973794 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:00:07.973741    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:00:08.076227 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:00:08.076185    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:23.712000 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:23.711965    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:26.300849 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:26.300757    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:28.584270 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:28.584230    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:28.683560 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:28.683525    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:28.742992 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:28.742963    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:31.660147 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:31.660115    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:31.734326 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:31.734288    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:51.516214 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:51.516174    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:51.655031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:51.654995    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:51.717543 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:51.717509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:01:52.221343 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:01:52.221309    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:02.266670 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:02.266616    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:02.372114 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:02.372066    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:02.474621 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:02.474586    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:12.319972 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:12.319929    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:12.415241 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:12.415208    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:17.481151 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:17.481115    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:17.770903 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:17.770814    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:17.968433 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:17.968398    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:18.142482 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:18.142446    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:18.315635 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:18.315598    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:18.447851 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:18.447729    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:18.531305 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:18.531269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:19.093495 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:19.093462    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:19.165533 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:19.165498    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:19.379603 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:19.379512    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:19.483004 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:19.482973    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:19.965938 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:19.965899    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:02:23.395913 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:02:23.395875    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:10.375168 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:10.375132    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:10.476215 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:10.476178    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:38.471977 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.471909    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:38.535315 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.535288    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-tkvrv"]
Sep 17 16:03:38.535514 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.535503    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f1d2f9-071d-41d5-b89b-21f756d328c4" containerName="registry"
Sep 17 16:03:38.535563 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.535516    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f1d2f9-071d-41d5-b89b-21f756d328c4" containerName="registry"
Sep 17 16:03:38.535563 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.535558    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f1d2f9-071d-41d5-b89b-21f756d328c4" containerName="registry"
Sep 17 16:03:38.538256 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.538237    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:38.540761 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.540734    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 16:03:38.540892 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.540748    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:03:38.541050 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.541038    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:03:38.541931 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.541916    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-9dvgt\""
Sep 17 16:03:38.544421 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.544395    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:38.550087 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.550068    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-tkvrv"]
Sep 17 16:03:38.603373 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.603338    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79b448c0-7017-4b88-b38c-dde693bfc86f-cert\") pod \"kserve-controller-manager-6fdd8844c6-tkvrv\" (UID: \"79b448c0-7017-4b88-b38c-dde693bfc86f\") " pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:38.603373 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.603374    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6kw4\" (UniqueName: \"kubernetes.io/projected/79b448c0-7017-4b88-b38c-dde693bfc86f-kube-api-access-l6kw4\") pod \"kserve-controller-manager-6fdd8844c6-tkvrv\" (UID: \"79b448c0-7017-4b88-b38c-dde693bfc86f\") " pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:38.652285 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.652259    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:38.704069 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.704042    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79b448c0-7017-4b88-b38c-dde693bfc86f-cert\") pod \"kserve-controller-manager-6fdd8844c6-tkvrv\" (UID: \"79b448c0-7017-4b88-b38c-dde693bfc86f\") " pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:38.704202 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.704075    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6kw4\" (UniqueName: \"kubernetes.io/projected/79b448c0-7017-4b88-b38c-dde693bfc86f-kube-api-access-l6kw4\") pod \"kserve-controller-manager-6fdd8844c6-tkvrv\" (UID: \"79b448c0-7017-4b88-b38c-dde693bfc86f\") " pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:38.706323 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.706295    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79b448c0-7017-4b88-b38c-dde693bfc86f-cert\") pod \"kserve-controller-manager-6fdd8844c6-tkvrv\" (UID: \"79b448c0-7017-4b88-b38c-dde693bfc86f\") " pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:38.714689 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.714669    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6kw4\" (UniqueName: \"kubernetes.io/projected/79b448c0-7017-4b88-b38c-dde693bfc86f-kube-api-access-l6kw4\") pod \"kserve-controller-manager-6fdd8844c6-tkvrv\" (UID: \"79b448c0-7017-4b88-b38c-dde693bfc86f\") " pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:38.751828 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.751759    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:38.847949 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.847915    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:38.855433 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:38.855406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:39.103459 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:39.103430    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:39.119918 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:39.119889    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-tkvrv"]
Sep 17 16:03:39.123103 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:03:39.123076    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b448c0_7017_4b88_b38c_dde693bfc86f.slice/crio-76ca1fc5e2f7d8e2e85e1e544d26ab20f241fdb2f034286138189c5a852daf56 WatchSource:0}: Error finding container 76ca1fc5e2f7d8e2e85e1e544d26ab20f241fdb2f034286138189c5a852daf56: Status 404 returned error can't find the container with id 76ca1fc5e2f7d8e2e85e1e544d26ab20f241fdb2f034286138189c5a852daf56
Sep 17 16:03:39.124515 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:39.124500    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:03:39.161734 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:39.161704    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:39.224172 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:39.224139    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:03:39.373013 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:39.372931    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv" event={"ID":"79b448c0-7017-4b88-b38c-dde693bfc86f","Type":"ContainerStarted","Data":"76ca1fc5e2f7d8e2e85e1e544d26ab20f241fdb2f034286138189c5a852daf56"}
Sep 17 16:03:43.389226 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:43.389187    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv" event={"ID":"79b448c0-7017-4b88-b38c-dde693bfc86f","Type":"ContainerStarted","Data":"783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8"}
Sep 17 16:03:43.389611 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:43.389335    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:03:43.407962 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:03:43.407915    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv" podStartSLOduration=2.020823853 podStartE2EDuration="5.407898756s" totalImagesPullingTime="3.387074903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:43.406684325 +0000 UTC m=+581.633353632" watchObservedRunningTime="2026-09-17 16:03:43.407898756 +0000 UTC m=+581.634568043"
Sep 17 16:04:14.397833 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:14.397800    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:04:22.692620 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.692582    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-tkvrv"]
Sep 17 16:04:22.695225 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.692828    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv" podUID="79b448c0-7017-4b88-b38c-dde693bfc86f" containerName="manager" containerID="cri-o://783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8" gracePeriod=10
Sep 17 16:04:22.719680 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.719089    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-t7rhk"]
Sep 17 16:04:22.723881 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.723848    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:22.731494 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.731471    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-t7rhk"]
Sep 17 16:04:22.800045 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.800014    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhclt\" (UniqueName: \"kubernetes.io/projected/5782e666-064b-4480-9e3a-1a38a1a3da63-kube-api-access-nhclt\") pod \"kserve-controller-manager-6fdd8844c6-t7rhk\" (UID: \"5782e666-064b-4480-9e3a-1a38a1a3da63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:22.800162 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.800059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5782e666-064b-4480-9e3a-1a38a1a3da63-cert\") pod \"kserve-controller-manager-6fdd8844c6-t7rhk\" (UID: \"5782e666-064b-4480-9e3a-1a38a1a3da63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:22.901002 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.900963    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhclt\" (UniqueName: \"kubernetes.io/projected/5782e666-064b-4480-9e3a-1a38a1a3da63-kube-api-access-nhclt\") pod \"kserve-controller-manager-6fdd8844c6-t7rhk\" (UID: \"5782e666-064b-4480-9e3a-1a38a1a3da63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:22.901002 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.901005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5782e666-064b-4480-9e3a-1a38a1a3da63-cert\") pod \"kserve-controller-manager-6fdd8844c6-t7rhk\" (UID: \"5782e666-064b-4480-9e3a-1a38a1a3da63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:22.903217 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.903189    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5782e666-064b-4480-9e3a-1a38a1a3da63-cert\") pod \"kserve-controller-manager-6fdd8844c6-t7rhk\" (UID: \"5782e666-064b-4480-9e3a-1a38a1a3da63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:22.910745 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.910716    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhclt\" (UniqueName: \"kubernetes.io/projected/5782e666-064b-4480-9e3a-1a38a1a3da63-kube-api-access-nhclt\") pod \"kserve-controller-manager-6fdd8844c6-t7rhk\" (UID: \"5782e666-064b-4480-9e3a-1a38a1a3da63\") " pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:22.934391 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:22.934372    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:04:23.002070 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.001987    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79b448c0-7017-4b88-b38c-dde693bfc86f-cert\") pod \"79b448c0-7017-4b88-b38c-dde693bfc86f\" (UID: \"79b448c0-7017-4b88-b38c-dde693bfc86f\") "
Sep 17 16:04:23.002070 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.002031    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6kw4\" (UniqueName: \"kubernetes.io/projected/79b448c0-7017-4b88-b38c-dde693bfc86f-kube-api-access-l6kw4\") pod \"79b448c0-7017-4b88-b38c-dde693bfc86f\" (UID: \"79b448c0-7017-4b88-b38c-dde693bfc86f\") "
Sep 17 16:04:23.004122 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.004088    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b448c0-7017-4b88-b38c-dde693bfc86f-cert" (OuterVolumeSpecName: "cert") pod "79b448c0-7017-4b88-b38c-dde693bfc86f" (UID: "79b448c0-7017-4b88-b38c-dde693bfc86f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:04:23.004256 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.004151    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b448c0-7017-4b88-b38c-dde693bfc86f-kube-api-access-l6kw4" (OuterVolumeSpecName: "kube-api-access-l6kw4") pod "79b448c0-7017-4b88-b38c-dde693bfc86f" (UID: "79b448c0-7017-4b88-b38c-dde693bfc86f"). InnerVolumeSpecName "kube-api-access-l6kw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:23.067242 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.067199    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:23.103331 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.103295    2584 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79b448c0-7017-4b88-b38c-dde693bfc86f-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:04:23.103331 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.103335    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6kw4\" (UniqueName: \"kubernetes.io/projected/79b448c0-7017-4b88-b38c-dde693bfc86f-kube-api-access-l6kw4\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:04:23.335917 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.335801    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-t7rhk"]
Sep 17 16:04:23.338406 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:04:23.338379    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5782e666_064b_4480_9e3a_1a38a1a3da63.slice/crio-5de96c9514457a7fcde06f151eff79fe68ecea5847c6a83e9624c2df27e8d2a8 WatchSource:0}: Error finding container 5de96c9514457a7fcde06f151eff79fe68ecea5847c6a83e9624c2df27e8d2a8: Status 404 returned error can't find the container with id 5de96c9514457a7fcde06f151eff79fe68ecea5847c6a83e9624c2df27e8d2a8
Sep 17 16:04:23.521542 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.521510    2584 generic.go:356] "Generic (PLEG): container finished" podID="79b448c0-7017-4b88-b38c-dde693bfc86f" containerID="783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8" exitCode=0
Sep 17 16:04:23.521711 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.521555    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv"
Sep 17 16:04:23.521711 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.521605    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv" event={"ID":"79b448c0-7017-4b88-b38c-dde693bfc86f","Type":"ContainerDied","Data":"783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8"}
Sep 17 16:04:23.521711 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.521660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-tkvrv" event={"ID":"79b448c0-7017-4b88-b38c-dde693bfc86f","Type":"ContainerDied","Data":"76ca1fc5e2f7d8e2e85e1e544d26ab20f241fdb2f034286138189c5a852daf56"}
Sep 17 16:04:23.521711 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.521682    2584 scope.go:117] "RemoveContainer" containerID="783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8"
Sep 17 16:04:23.522896 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.522867    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk" event={"ID":"5782e666-064b-4480-9e3a-1a38a1a3da63","Type":"ContainerStarted","Data":"5de96c9514457a7fcde06f151eff79fe68ecea5847c6a83e9624c2df27e8d2a8"}
Sep 17 16:04:23.529834 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.529817    2584 scope.go:117] "RemoveContainer" containerID="783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8"
Sep 17 16:04:23.530095 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:04:23.530074    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8\": container with ID starting with 783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8 not found: ID does not exist" containerID="783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8"
Sep 17 16:04:23.530139 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.530103    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8"} err="failed to get container status \"783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8\": rpc error: code = NotFound desc = could not find container \"783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8\": container with ID starting with 783120eeea4dea94b14e2820f59affee702776a44c9ba0280d7188c0f47cd7c8 not found: ID does not exist"
Sep 17 16:04:23.548564 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.548537    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-tkvrv"]
Sep 17 16:04:23.557964 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:23.557936    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6fdd8844c6-tkvrv"]
Sep 17 16:04:24.370969 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:24.370941    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b448c0-7017-4b88-b38c-dde693bfc86f" path="/var/lib/kubelet/pods/79b448c0-7017-4b88-b38c-dde693bfc86f/volumes"
Sep 17 16:04:24.527672 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:24.527636    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk" event={"ID":"5782e666-064b-4480-9e3a-1a38a1a3da63","Type":"ContainerStarted","Data":"290312a32dd0d8efeaa3b92fe403a8733ff85e90cf4c03d3f2fad358f8377379"}
Sep 17 16:04:24.527869 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:24.527708    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:24.543671 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:24.543620    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk" podStartSLOduration=2.085893331 podStartE2EDuration="2.543606197s" totalImagesPullingTime="457.712866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:04:24.5435323 +0000 UTC m=+622.770201601" watchObservedRunningTime="2026-09-17 16:04:24.543606197 +0000 UTC m=+622.770275485"
Sep 17 16:04:55.536611 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:55.536574    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6fdd8844c6-t7rhk"
Sep 17 16:04:56.950591 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.950551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:56.984111 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.984082    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-d4sf4"]
Sep 17 16:04:56.984316 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.984304    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b448c0-7017-4b88-b38c-dde693bfc86f" containerName="manager"
Sep 17 16:04:56.984364 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.984317    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b448c0-7017-4b88-b38c-dde693bfc86f" containerName="manager"
Sep 17 16:04:56.984397 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.984363    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b448c0-7017-4b88-b38c-dde693bfc86f" containerName="manager"
Sep 17 16:04:56.987292 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.987267    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:04:56.990161 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.990140    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-rt8pt\""
Sep 17 16:04:56.990284 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.990188    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 16:04:56.994633 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.994611    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-tfcvd"]
Sep 17 16:04:56.997385 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:56.997369    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.001041 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.001021    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 16:04:57.001351 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.001338    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-2frr8\""
Sep 17 16:04:57.002101 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.002083    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 16:04:57.006579 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.006559    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-d4sf4"]
Sep 17 16:04:57.017711 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.017687    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-tfcvd"]
Sep 17 16:04:57.046012 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.045983    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:57.134887 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.134850    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-cert\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.134887 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.134892    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76210352-393c-4666-8fa7-1e4930d11d1f-tls-certs\") pod \"model-serving-api-7b954ffc95-d4sf4\" (UID: \"76210352-393c-4666-8fa7-1e4930d11d1f\") " pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:04:57.135087 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.134950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm4h\" (UniqueName: \"kubernetes.io/projected/960d461f-a29a-4ee3-b690-6ed3248c5203-kube-api-access-6gm4h\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.135087 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.134987    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7v5\" (UniqueName: \"kubernetes.io/projected/76210352-393c-4666-8fa7-1e4930d11d1f-kube-api-access-zc7v5\") pod \"model-serving-api-7b954ffc95-d4sf4\" (UID: \"76210352-393c-4666-8fa7-1e4930d11d1f\") " pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:04:57.135087 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.135012    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-metrics-certs\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.174146 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.174106    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:57.236020 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.235949    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-cert\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.236020 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.235989    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76210352-393c-4666-8fa7-1e4930d11d1f-tls-certs\") pod \"model-serving-api-7b954ffc95-d4sf4\" (UID: \"76210352-393c-4666-8fa7-1e4930d11d1f\") " pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:04:57.236020 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.236012    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm4h\" (UniqueName: \"kubernetes.io/projected/960d461f-a29a-4ee3-b690-6ed3248c5203-kube-api-access-6gm4h\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.236232 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.236032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7v5\" (UniqueName: \"kubernetes.io/projected/76210352-393c-4666-8fa7-1e4930d11d1f-kube-api-access-zc7v5\") pod \"model-serving-api-7b954ffc95-d4sf4\" (UID: \"76210352-393c-4666-8fa7-1e4930d11d1f\") " pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:04:57.236232 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.236053    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-metrics-certs\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.236232 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:04:57.236104    2584 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 16:04:57.236232 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:04:57.236181    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-cert podName:960d461f-a29a-4ee3-b690-6ed3248c5203 nodeName:}" failed. No retries permitted until 2026-09-17 16:04:57.736162857 +0000 UTC m=+655.962832132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-cert") pod "odh-model-controller-7b966d7984-tfcvd" (UID: "960d461f-a29a-4ee3-b690-6ed3248c5203") : secret "odh-model-controller-webhook-cert" not found
Sep 17 16:04:57.238586 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.238552    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-metrics-certs\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.238710 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.238559    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76210352-393c-4666-8fa7-1e4930d11d1f-tls-certs\") pod \"model-serving-api-7b954ffc95-d4sf4\" (UID: \"76210352-393c-4666-8fa7-1e4930d11d1f\") " pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:04:57.251206 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.251187    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7v5\" (UniqueName: \"kubernetes.io/projected/76210352-393c-4666-8fa7-1e4930d11d1f-kube-api-access-zc7v5\") pod \"model-serving-api-7b954ffc95-d4sf4\" (UID: \"76210352-393c-4666-8fa7-1e4930d11d1f\") " pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:04:57.256573 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.256553    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm4h\" (UniqueName: \"kubernetes.io/projected/960d461f-a29a-4ee3-b690-6ed3248c5203-kube-api-access-6gm4h\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.273962 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.273937    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:57.296659 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.296637    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:04:57.378012 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.377982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:57.565679 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.565655    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-d4sf4"]
Sep 17 16:04:57.568423 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:04:57.568401    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76210352_393c_4666_8fa7_1e4930d11d1f.slice/crio-0e68c35365cf459ba6759f2967da9d8ee2d19085816898b65d4a17a0608cd854 WatchSource:0}: Error finding container 0e68c35365cf459ba6759f2967da9d8ee2d19085816898b65d4a17a0608cd854: Status 404 returned error can't find the container with id 0e68c35365cf459ba6759f2967da9d8ee2d19085816898b65d4a17a0608cd854
Sep 17 16:04:57.633426 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.633394    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-d4sf4" event={"ID":"76210352-393c-4666-8fa7-1e4930d11d1f","Type":"ContainerStarted","Data":"0e68c35365cf459ba6759f2967da9d8ee2d19085816898b65d4a17a0608cd854"}
Sep 17 16:04:57.740746 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.740706    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-cert\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.743044 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.743021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/960d461f-a29a-4ee3-b690-6ed3248c5203-cert\") pod \"odh-model-controller-7b966d7984-tfcvd\" (UID: \"960d461f-a29a-4ee3-b690-6ed3248c5203\") " pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:57.907025 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:57.906943    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:04:58.179372 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:58.179290    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-tfcvd"]
Sep 17 16:04:58.182327 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:04:58.182291    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960d461f_a29a_4ee3_b690_6ed3248c5203.slice/crio-d775251fd9d5fb6f1749992387aca27da5e0ed131bdfde98d1c5a8008ea02589 WatchSource:0}: Error finding container d775251fd9d5fb6f1749992387aca27da5e0ed131bdfde98d1c5a8008ea02589: Status 404 returned error can't find the container with id d775251fd9d5fb6f1749992387aca27da5e0ed131bdfde98d1c5a8008ea02589
Sep 17 16:04:58.453399 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:58.453269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:58.551577 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:58.551528    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:04:58.644692 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:04:58.644657    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-tfcvd" event={"ID":"960d461f-a29a-4ee3-b690-6ed3248c5203","Type":"ContainerStarted","Data":"d775251fd9d5fb6f1749992387aca27da5e0ed131bdfde98d1c5a8008ea02589"}
Sep 17 16:05:01.659317 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:05:01.659275    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-d4sf4" event={"ID":"76210352-393c-4666-8fa7-1e4930d11d1f","Type":"ContainerStarted","Data":"473847ae3a27bf224bc1951abd57ec56eb6c4cf4b6e2c3c891ffa1a430c6fc44"}
Sep 17 16:05:01.659758 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:05:01.659378    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:05:01.660844 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:05:01.660818    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-tfcvd" event={"ID":"960d461f-a29a-4ee3-b690-6ed3248c5203","Type":"ContainerStarted","Data":"fec6e3f3af3d6ceaa45f056c74dad2bf8f166834f85698f20154d4844ab6e3b7"}
Sep 17 16:05:01.660973 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:05:01.660957    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:05:01.676798 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:05:01.676730    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-d4sf4" podStartSLOduration=2.048443644 podStartE2EDuration="5.676713292s" totalImagesPullingTime="3.628269648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:01.675835484 +0000 UTC m=+659.902504811" watchObservedRunningTime="2026-09-17 16:05:01.676713292 +0000 UTC m=+659.903382579"
Sep 17 16:05:01.691820 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:05:01.691755    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-tfcvd" podStartSLOduration=2.63529128 podStartE2EDuration="5.691739619s" totalImagesPullingTime="3.056448339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:01.691348022 +0000 UTC m=+659.918017310" watchObservedRunningTime="2026-09-17 16:05:01.691739619 +0000 UTC m=+659.918408906"
Sep 17 16:05:12.667043 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:05:12.667012    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-tfcvd"
Sep 17 16:05:12.668944 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:05:12.668923    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-d4sf4"
Sep 17 16:06:06.083172 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:06.083135    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:32.921951 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:32.921912    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:34.811077 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:34.811042    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:44.718339 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.718296    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:44.826712 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.826675    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:44.846667 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.846630    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"]
Sep 17 16:06:44.850031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.850006    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:44.852611 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.852579    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:06:44.852759 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.852676    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:06:44.853889 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.853853    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z8dwv\""
Sep 17 16:06:44.853889 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.853862    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:06:44.854058 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.853895    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-ba470-kube-rbac-proxy-sar-config\""
Sep 17 16:06:44.854058 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.853862    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-ba470-predictor-serving-cert\""
Sep 17 16:06:44.854058 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.853928    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:06:44.862955 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.862929    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"]
Sep 17 16:06:44.928931 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:44.928824    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:45.004444 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.004360    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:45.022381 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.022348    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/954a3e80-2531-4339-bb8e-cace724d0a92-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.022540 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.022390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.022540 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.022423    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lkw\" (UniqueName: \"kubernetes.io/projected/954a3e80-2531-4339-bb8e-cace724d0a92-kube-api-access-l4lkw\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.022540 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.022511    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/954a3e80-2531-4339-bb8e-cace724d0a92-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.022656 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.022536    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.099977 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.099947    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"]
Sep 17 16:06:45.103191 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.103175    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.105552 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.105531    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 16:06:45.105730 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.105710    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 16:06:45.116315 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.116293    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"]
Sep 17 16:06:45.123027 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.123002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lkw\" (UniqueName: \"kubernetes.io/projected/954a3e80-2531-4339-bb8e-cace724d0a92-kube-api-access-l4lkw\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.123141 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.123061    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/954a3e80-2531-4339-bb8e-cace724d0a92-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.123261 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.123237    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.123337 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.123319    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/954a3e80-2531-4339-bb8e-cace724d0a92-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.123394 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.123376    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.123713 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.123691    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/954a3e80-2531-4339-bb8e-cace724d0a92-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.123965 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.123940    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.124110 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.124089    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.125851 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.125827    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/954a3e80-2531-4339-bb8e-cace724d0a92-proxy-tls\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.131541 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.131497    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lkw\" (UniqueName: \"kubernetes.io/projected/954a3e80-2531-4339-bb8e-cace724d0a92-kube-api-access-l4lkw\") pod \"isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.136123 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.135904    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:45.160103 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.160072    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:06:45.224351 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.224319    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f2f10-c0f7-421f-a830-f1664af854c7-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.224497 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.224388    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.224497 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.224426    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12f2f10-c0f7-421f-a830-f1664af854c7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.224497 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.224460    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzb6\" (UniqueName: \"kubernetes.io/projected/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-api-access-2hzb6\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.224497 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.224490    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.252187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.252128    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:06:45.325193 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.325156    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f2f10-c0f7-421f-a830-f1664af854c7-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.325370 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.325213    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.325370 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.325243    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12f2f10-c0f7-421f-a830-f1664af854c7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.325370 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.325265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzb6\" (UniqueName: \"kubernetes.io/projected/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-api-access-2hzb6\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.325370 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.325287    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.325761 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.325731    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12f2f10-c0f7-421f-a830-f1664af854c7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.326054 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.326035    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.326216 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.326195    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.327851 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.327826    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f2f10-c0f7-421f-a830-f1664af854c7-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.333817 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.333777    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzb6\" (UniqueName: \"kubernetes.io/projected/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-api-access-2hzb6\") pod \"isvc-canary-rollback-predictor-6b9dfd44-wncxp\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.413358 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.413323    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:06:45.437591 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.437363    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"]
Sep 17 16:06:45.440759 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:06:45.440720    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954a3e80_2531_4339_bb8e_cace724d0a92.slice/crio-51de72610512495ac88c1f5d3092d92d205da29623ac05943bad651e9d5bd638 WatchSource:0}: Error finding container 51de72610512495ac88c1f5d3092d92d205da29623ac05943bad651e9d5bd638: Status 404 returned error can't find the container with id 51de72610512495ac88c1f5d3092d92d205da29623ac05943bad651e9d5bd638
Sep 17 16:06:45.687929 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:45.687881    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"]
Sep 17 16:06:45.690316 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:06:45.690284    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12f2f10_c0f7_421f_a830_f1664af854c7.slice/crio-6ee6681d164992132bdfba880f9339e86927b70127f83fb567d90f7877ed35d3 WatchSource:0}: Error finding container 6ee6681d164992132bdfba880f9339e86927b70127f83fb567d90f7877ed35d3: Status 404 returned error can't find the container with id 6ee6681d164992132bdfba880f9339e86927b70127f83fb567d90f7877ed35d3
Sep 17 16:06:46.008411 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:46.008325    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerStarted","Data":"51de72610512495ac88c1f5d3092d92d205da29623ac05943bad651e9d5bd638"}
Sep 17 16:06:46.009374 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:46.009353    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" event={"ID":"d12f2f10-c0f7-421f-a830-f1664af854c7","Type":"ContainerStarted","Data":"6ee6681d164992132bdfba880f9339e86927b70127f83fb567d90f7877ed35d3"}
Sep 17 16:06:49.024338 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:49.024295    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerStarted","Data":"5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf"}
Sep 17 16:06:49.026208 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:49.026173    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" event={"ID":"d12f2f10-c0f7-421f-a830-f1664af854c7","Type":"ContainerStarted","Data":"943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a"}
Sep 17 16:06:53.044127 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:53.044041    2584 generic.go:356] "Generic (PLEG): container finished" podID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerID="943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a" exitCode=0
Sep 17 16:06:53.044127 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:53.044114    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" event={"ID":"d12f2f10-c0f7-421f-a830-f1664af854c7","Type":"ContainerDied","Data":"943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a"}
Sep 17 16:06:53.045895 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:53.045871    2584 generic.go:356] "Generic (PLEG): container finished" podID="954a3e80-2531-4339-bb8e-cace724d0a92" containerID="5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf" exitCode=0
Sep 17 16:06:53.045967 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:06:53.045922    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerDied","Data":"5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf"}
Sep 17 16:07:11.155300 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:11.155255    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" event={"ID":"d12f2f10-c0f7-421f-a830-f1664af854c7","Type":"ContainerStarted","Data":"c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5"}
Sep 17 16:07:11.160258 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:11.160195    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerStarted","Data":"88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511"}
Sep 17 16:07:13.170820 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:13.170763    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" event={"ID":"d12f2f10-c0f7-421f-a830-f1664af854c7","Type":"ContainerStarted","Data":"514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2"}
Sep 17 16:07:13.171393 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:13.170904    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:07:13.171393 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:13.170939    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:07:13.172342 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:13.172308    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:07:13.173147 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:13.173126    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerStarted","Data":"cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf"}
Sep 17 16:07:13.193088 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:13.193041    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podStartSLOduration=1.6677113430000001 podStartE2EDuration="28.193027251s" totalImagesPullingTime="22.70246882s" totalInitContainerRuntime="3.822847088s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:13.192549185 +0000 UTC m=+791.419218501" watchObservedRunningTime="2026-09-17 16:07:13.193027251 +0000 UTC m=+791.419696550"
Sep 17 16:07:13.965188 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:13.965153    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:14.177378 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:14.177341    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:07:15.183224 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:15.183190    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerStarted","Data":"0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de"}
Sep 17 16:07:15.183587 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:15.183297    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:07:16.187070 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:16.187039    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:07:16.187462 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:16.187113    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:07:16.188390 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:16.188359    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:07:16.189091 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:16.189070    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:17.190864 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:17.190820    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:07:17.191272 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:17.191190    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:17.194535 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:17.194515    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:07:17.235524 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:17.235479    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podStartSLOduration=4.41937934 podStartE2EDuration="33.235461021s" totalImagesPullingTime="24.979362237s" totalInitContainerRuntime="3.836719444s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:44 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:15.214440124 +0000 UTC m=+793.441109412" watchObservedRunningTime="2026-09-17 16:07:17.235461021 +0000 UTC m=+795.462130311"
Sep 17 16:07:18.194133 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:18.194084    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:07:18.194543 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:18.194360    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:19.182223 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:19.182194    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:07:19.182738 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:19.182716    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:07:28.194343 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:28.194249    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:07:28.194716 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:28.194694    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:29.183486 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:29.183452    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:07:38.194243 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:38.194191    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:07:38.194708 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:38.194556    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:39.183663 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:39.183617    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:07:48.194727 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:48.194680    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:07:48.195196 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:48.195150    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:49.182923 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:49.182887    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:07:52.459985 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:52.459947    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:07:58.194068 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:58.194017    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:07:58.194490 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:58.194414    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:07:59.183489 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:07:59.183450    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:08:08.194738 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:08.194689    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:08:08.195211 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:08.195181    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:09.183617 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:09.183580    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:08:18.194472 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:18.194414    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:08:18.194937 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:18.194835    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:19.183473 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:19.183432    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 17 16:08:25.791717 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:25.791681    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"]
Sep 17 16:08:25.792189 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:25.792047    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kube-rbac-proxy" containerID="cri-o://514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2" gracePeriod=30
Sep 17 16:08:25.792189 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:25.792032    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container" containerID="cri-o://c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5" gracePeriod=30
Sep 17 16:08:25.793073 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:25.793056    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:08:26.448195 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:26.448158    2584 generic.go:356] "Generic (PLEG): container finished" podID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerID="514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2" exitCode=2
Sep 17 16:08:26.448353 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:26.448226    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" event={"ID":"d12f2f10-c0f7-421f-a830-f1664af854c7","Type":"ContainerDied","Data":"514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2"}
Sep 17 16:08:28.194983 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:28.194946    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:08:28.195478 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:28.195443    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:08:29.177970 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:29.177927    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 17 16:08:30.423611 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.423591    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:08:30.465322 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.465287    2584 generic.go:356] "Generic (PLEG): container finished" podID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerID="c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5" exitCode=0
Sep 17 16:08:30.465474 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.465353    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"
Sep 17 16:08:30.465474 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.465370    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" event={"ID":"d12f2f10-c0f7-421f-a830-f1664af854c7","Type":"ContainerDied","Data":"c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5"}
Sep 17 16:08:30.465474 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.465432    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp" event={"ID":"d12f2f10-c0f7-421f-a830-f1664af854c7","Type":"ContainerDied","Data":"6ee6681d164992132bdfba880f9339e86927b70127f83fb567d90f7877ed35d3"}
Sep 17 16:08:30.465474 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.465461    2584 scope.go:117] "RemoveContainer" containerID="514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2"
Sep 17 16:08:30.474032 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.474015    2584 scope.go:117] "RemoveContainer" containerID="c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5"
Sep 17 16:08:30.480824 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.480809    2584 scope.go:117] "RemoveContainer" containerID="943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a"
Sep 17 16:08:30.487497 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.487481    2584 scope.go:117] "RemoveContainer" containerID="514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2"
Sep 17 16:08:30.487751 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:08:30.487734    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2\": container with ID starting with 514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2 not found: ID does not exist" containerID="514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2"
Sep 17 16:08:30.487892 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.487770    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2"} err="failed to get container status \"514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2\": rpc error: code = NotFound desc = could not find container \"514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2\": container with ID starting with 514d24c00b8b28c0c2e5eb1a1b0dfb7ed581a601f643af70a523e86e1cdb59b2 not found: ID does not exist"
Sep 17 16:08:30.487892 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.487807    2584 scope.go:117] "RemoveContainer" containerID="c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5"
Sep 17 16:08:30.488100 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:08:30.488081    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5\": container with ID starting with c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5 not found: ID does not exist" containerID="c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5"
Sep 17 16:08:30.488143 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.488108    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5"} err="failed to get container status \"c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5\": rpc error: code = NotFound desc = could not find container \"c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5\": container with ID starting with c796620312f01569aa5f2ef603b2e95fa35e2a16a0dba563bd14a28aacf6ffb5 not found: ID does not exist"
Sep 17 16:08:30.488143 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.488124    2584 scope.go:117] "RemoveContainer" containerID="943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a"
Sep 17 16:08:30.488365 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:08:30.488347    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a\": container with ID starting with 943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a not found: ID does not exist" containerID="943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a"
Sep 17 16:08:30.488403 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.488370    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a"} err="failed to get container status \"943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a\": rpc error: code = NotFound desc = could not find container \"943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a\": container with ID starting with 943d313e8cd4c0711c833094002d742c8860c34e6b3b08833976a0c81108933a not found: ID does not exist"
Sep 17 16:08:30.533426 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.533399    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f2f10-c0f7-421f-a830-f1664af854c7-proxy-tls\") pod \"d12f2f10-c0f7-421f-a830-f1664af854c7\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") "
Sep 17 16:08:30.533580 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.533460    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-cabundle-cert\") pod \"d12f2f10-c0f7-421f-a830-f1664af854c7\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") "
Sep 17 16:08:30.533580 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.533498    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hzb6\" (UniqueName: \"kubernetes.io/projected/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-api-access-2hzb6\") pod \"d12f2f10-c0f7-421f-a830-f1664af854c7\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") "
Sep 17 16:08:30.533580 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.533523    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-rbac-proxy-sar-config\") pod \"d12f2f10-c0f7-421f-a830-f1664af854c7\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") "
Sep 17 16:08:30.533580 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.533556    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12f2f10-c0f7-421f-a830-f1664af854c7-kserve-provision-location\") pod \"d12f2f10-c0f7-421f-a830-f1664af854c7\" (UID: \"d12f2f10-c0f7-421f-a830-f1664af854c7\") "
Sep 17 16:08:30.534000 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.533952    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d12f2f10-c0f7-421f-a830-f1664af854c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d12f2f10-c0f7-421f-a830-f1664af854c7" (UID: "d12f2f10-c0f7-421f-a830-f1664af854c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:30.534125 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.533992    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d12f2f10-c0f7-421f-a830-f1664af854c7" (UID: "d12f2f10-c0f7-421f-a830-f1664af854c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:30.534125 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.534044    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d12f2f10-c0f7-421f-a830-f1664af854c7" (UID: "d12f2f10-c0f7-421f-a830-f1664af854c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:30.535715 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.535691    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-api-access-2hzb6" (OuterVolumeSpecName: "kube-api-access-2hzb6") pod "d12f2f10-c0f7-421f-a830-f1664af854c7" (UID: "d12f2f10-c0f7-421f-a830-f1664af854c7"). InnerVolumeSpecName "kube-api-access-2hzb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:30.535850 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.535806    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12f2f10-c0f7-421f-a830-f1664af854c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d12f2f10-c0f7-421f-a830-f1664af854c7" (UID: "d12f2f10-c0f7-421f-a830-f1664af854c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:30.634352 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.634312    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:08:30.634352 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.634343    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hzb6\" (UniqueName: \"kubernetes.io/projected/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-api-access-2hzb6\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:08:30.634352 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.634353    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d12f2f10-c0f7-421f-a830-f1664af854c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:08:30.634352 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.634363    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d12f2f10-c0f7-421f-a830-f1664af854c7-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:08:30.634625 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.634373    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f2f10-c0f7-421f-a830-f1664af854c7-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:08:30.788793 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.788754    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"]
Sep 17 16:08:30.794357 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:30.794333    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-wncxp"]
Sep 17 16:08:32.370156 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:32.370125    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" path="/var/lib/kubelet/pods/d12f2f10-c0f7-421f-a830-f1664af854c7/volumes"
Sep 17 16:08:34.741263 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.741229    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:34.782105 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782080    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"]
Sep 17 16:08:34.782328 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782316    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="storage-initializer"
Sep 17 16:08:34.782375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782329    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="storage-initializer"
Sep 17 16:08:34.782375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782337    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kube-rbac-proxy"
Sep 17 16:08:34.782375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782343    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kube-rbac-proxy"
Sep 17 16:08:34.782375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782355    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container"
Sep 17 16:08:34.782375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782360    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container"
Sep 17 16:08:34.782526 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782410    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kube-rbac-proxy"
Sep 17 16:08:34.782526 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.782418    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12f2f10-c0f7-421f-a830-f1664af854c7" containerName="kserve-container"
Sep 17 16:08:34.786815 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.786774    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.789205 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.789183    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 16:08:34.789289 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.789236    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 16:08:34.796037 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.796017    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"]
Sep 17 16:08:34.864704 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.864669    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c32301-34fc-464c-a8f4-62593e3d41ef-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.864704 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.864708    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c32301-34fc-464c-a8f4-62593e3d41ef-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.864945 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.864740    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qp58\" (UniqueName: \"kubernetes.io/projected/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-api-access-4qp58\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.864945 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.864808    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.864945 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.864853    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.965742 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.965700    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c32301-34fc-464c-a8f4-62593e3d41ef-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.965949 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.965756    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qp58\" (UniqueName: \"kubernetes.io/projected/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-api-access-4qp58\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.965949 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.965794    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.965949 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.965818    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.965949 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.965871    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c32301-34fc-464c-a8f4-62593e3d41ef-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.966347 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.966327    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c32301-34fc-464c-a8f4-62593e3d41ef-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.966605 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.966587    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.966644 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.966590    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.968172 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.968148    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c32301-34fc-464c-a8f4-62593e3d41ef-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:34.974566 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:34.974534    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qp58\" (UniqueName: \"kubernetes.io/projected/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-api-access-4qp58\") pod \"isvc-canary-rollback-predictor-6b9dfd44-4n79g\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:35.096705 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:35.096660    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:35.370755 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:35.370595    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"]
Sep 17 16:08:35.373324 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:08:35.373297    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c32301_34fc_464c_a8f4_62593e3d41ef.slice/crio-933f55bcc10e81728ba0917eeec4850a813917475a3586825601a8e7ee7218ba WatchSource:0}: Error finding container 933f55bcc10e81728ba0917eeec4850a813917475a3586825601a8e7ee7218ba: Status 404 returned error can't find the container with id 933f55bcc10e81728ba0917eeec4850a813917475a3586825601a8e7ee7218ba
Sep 17 16:08:35.484731 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:35.484692    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" event={"ID":"43c32301-34fc-464c-a8f4-62593e3d41ef","Type":"ContainerStarted","Data":"d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a"}
Sep 17 16:08:35.484914 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:35.484751    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" event={"ID":"43c32301-34fc-464c-a8f4-62593e3d41ef","Type":"ContainerStarted","Data":"933f55bcc10e81728ba0917eeec4850a813917475a3586825601a8e7ee7218ba"}
Sep 17 16:08:39.505880 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:39.505797    2584 generic.go:356] "Generic (PLEG): container finished" podID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerID="d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a" exitCode=0
Sep 17 16:08:39.505880 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:39.505865    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" event={"ID":"43c32301-34fc-464c-a8f4-62593e3d41ef","Type":"ContainerDied","Data":"d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a"}
Sep 17 16:08:39.758212 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:39.758125    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"]
Sep 17 16:08:39.758562 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:39.758431    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" containerID="cri-o://0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de" gracePeriod=30
Sep 17 16:08:39.758562 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:39.758428    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" containerID="cri-o://88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511" gracePeriod=30
Sep 17 16:08:39.758562 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:39.758466    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy" containerID="cri-o://cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf" gracePeriod=30
Sep 17 16:08:39.781677 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:39.781650    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:39.880632 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:39.880551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:40.015632 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.015551    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:40.240273 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.240235    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:40.456052 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.456016    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:08:40.511697 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.511660    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" event={"ID":"43c32301-34fc-464c-a8f4-62593e3d41ef","Type":"ContainerStarted","Data":"98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1"}
Sep 17 16:08:40.512166 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.511738    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" event={"ID":"43c32301-34fc-464c-a8f4-62593e3d41ef","Type":"ContainerStarted","Data":"1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0"}
Sep 17 16:08:40.512166 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.511965    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:40.514213 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.514191    2584 generic.go:356] "Generic (PLEG): container finished" podID="954a3e80-2531-4339-bb8e-cace724d0a92" containerID="cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf" exitCode=2
Sep 17 16:08:40.514310 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.514257    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerDied","Data":"cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf"}
Sep 17 16:08:40.534912 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:40.534858    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podStartSLOduration=2.763778054 podStartE2EDuration="6.534841258s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771063204s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:40.533432376 +0000 UTC m=+878.760101726" watchObservedRunningTime="2026-09-17 16:08:40.534841258 +0000 UTC m=+878.761510546"
Sep 17 16:08:41.521164 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:41.521130    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:41.522219 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:41.522187    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:08:42.191284 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:42.191243    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 17 16:08:42.525040 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:42.524951    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:08:45.538916 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:45.538879    2584 generic.go:356] "Generic (PLEG): container finished" podID="954a3e80-2531-4339-bb8e-cace724d0a92" containerID="88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511" exitCode=0
Sep 17 16:08:45.539285 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:45.538935    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerDied","Data":"88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511"}
Sep 17 16:08:47.191470 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:47.191422    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 17 16:08:47.530101 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:47.530026    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:08:47.530532 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:47.530503    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:08:48.195052 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:48.195009    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:08:48.195856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:48.195826    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:08:52.190849 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:52.190810    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 17 16:08:52.191319 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:52.190948    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:08:57.191963 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:57.191871    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 17 16:08:57.531351 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:57.531262    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:08:58.194904 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:58.194852    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:08:58.196515 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:08:58.196483    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:02.191742 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:02.191707    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 17 16:09:07.191662 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:07.191610    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 17 16:09:07.530901 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:07.530813    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:09:08.194621 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:08.194581    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 17 16:09:08.195012 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:08.194706    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:09:08.196205 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:08.196176    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:09:08.196324 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:08.196275    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:09:09.901079 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:09.901052    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:09:10.031451 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.031368    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/954a3e80-2531-4339-bb8e-cace724d0a92-kserve-provision-location\") pod \"954a3e80-2531-4339-bb8e-cace724d0a92\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") "
Sep 17 16:09:10.031451 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.031451    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-cabundle-cert\") pod \"954a3e80-2531-4339-bb8e-cace724d0a92\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") "
Sep 17 16:09:10.031631 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.031469    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/954a3e80-2531-4339-bb8e-cace724d0a92-proxy-tls\") pod \"954a3e80-2531-4339-bb8e-cace724d0a92\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") "
Sep 17 16:09:10.031631 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.031500    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-kube-rbac-proxy-sar-config\") pod \"954a3e80-2531-4339-bb8e-cace724d0a92\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") "
Sep 17 16:09:10.031631 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.031530    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4lkw\" (UniqueName: \"kubernetes.io/projected/954a3e80-2531-4339-bb8e-cace724d0a92-kube-api-access-l4lkw\") pod \"954a3e80-2531-4339-bb8e-cace724d0a92\" (UID: \"954a3e80-2531-4339-bb8e-cace724d0a92\") "
Sep 17 16:09:10.031830 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.031720    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954a3e80-2531-4339-bb8e-cace724d0a92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "954a3e80-2531-4339-bb8e-cace724d0a92" (UID: "954a3e80-2531-4339-bb8e-cace724d0a92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:09:10.031882 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.031862    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "954a3e80-2531-4339-bb8e-cace724d0a92" (UID: "954a3e80-2531-4339-bb8e-cace724d0a92"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:09:10.031920 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.031873    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "954a3e80-2531-4339-bb8e-cace724d0a92" (UID: "954a3e80-2531-4339-bb8e-cace724d0a92"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:09:10.033483 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.033461    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/954a3e80-2531-4339-bb8e-cace724d0a92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "954a3e80-2531-4339-bb8e-cace724d0a92" (UID: "954a3e80-2531-4339-bb8e-cace724d0a92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:09:10.033571 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.033510    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954a3e80-2531-4339-bb8e-cace724d0a92-kube-api-access-l4lkw" (OuterVolumeSpecName: "kube-api-access-l4lkw") pod "954a3e80-2531-4339-bb8e-cace724d0a92" (UID: "954a3e80-2531-4339-bb8e-cace724d0a92"). InnerVolumeSpecName "kube-api-access-l4lkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:09:10.133029 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.132983    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:09:10.133029 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.133022    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/954a3e80-2531-4339-bb8e-cace724d0a92-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:09:10.133029 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.133032    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/954a3e80-2531-4339-bb8e-cace724d0a92-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:09:10.133029 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.133043    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4lkw\" (UniqueName: \"kubernetes.io/projected/954a3e80-2531-4339-bb8e-cace724d0a92-kube-api-access-l4lkw\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:09:10.133318 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.133052    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/954a3e80-2531-4339-bb8e-cace724d0a92-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:09:10.629117 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.629081    2584 generic.go:356] "Generic (PLEG): container finished" podID="954a3e80-2531-4339-bb8e-cace724d0a92" containerID="0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de" exitCode=0
Sep 17 16:09:10.629282 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.629137    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerDied","Data":"0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de"}
Sep 17 16:09:10.629282 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.629164    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"
Sep 17 16:09:10.629282 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.629215    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq" event={"ID":"954a3e80-2531-4339-bb8e-cace724d0a92","Type":"ContainerDied","Data":"51de72610512495ac88c1f5d3092d92d205da29623ac05943bad651e9d5bd638"}
Sep 17 16:09:10.629282 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.629246    2584 scope.go:117] "RemoveContainer" containerID="0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de"
Sep 17 16:09:10.638462 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.638443    2584 scope.go:117] "RemoveContainer" containerID="cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf"
Sep 17 16:09:10.645118 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.645100    2584 scope.go:117] "RemoveContainer" containerID="88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511"
Sep 17 16:09:10.649198 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.649178    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"]
Sep 17 16:09:10.652081 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.652045    2584 scope.go:117] "RemoveContainer" containerID="5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf"
Sep 17 16:09:10.653824 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.653804    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-ba470-predictor-7b5c8dff7c-s8pnq"]
Sep 17 16:09:10.659141 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.659119    2584 scope.go:117] "RemoveContainer" containerID="0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de"
Sep 17 16:09:10.659387 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:09:10.659368    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de\": container with ID starting with 0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de not found: ID does not exist" containerID="0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de"
Sep 17 16:09:10.659431 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.659395    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de"} err="failed to get container status \"0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de\": rpc error: code = NotFound desc = could not find container \"0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de\": container with ID starting with 0600a0ed3c91d835393fb88e29df9965d1ca80f1079fea04f404ab05ad8338de not found: ID does not exist"
Sep 17 16:09:10.659431 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.659413    2584 scope.go:117] "RemoveContainer" containerID="cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf"
Sep 17 16:09:10.659630 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:09:10.659614    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf\": container with ID starting with cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf not found: ID does not exist" containerID="cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf"
Sep 17 16:09:10.659709 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.659631    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf"} err="failed to get container status \"cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf\": rpc error: code = NotFound desc = could not find container \"cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf\": container with ID starting with cb5805350517214e450c05c5e82c736c5990465243ddff89b11588c227411ebf not found: ID does not exist"
Sep 17 16:09:10.659709 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.659641    2584 scope.go:117] "RemoveContainer" containerID="88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511"
Sep 17 16:09:10.659940 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:09:10.659924    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511\": container with ID starting with 88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511 not found: ID does not exist" containerID="88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511"
Sep 17 16:09:10.659989 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.659943    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511"} err="failed to get container status \"88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511\": rpc error: code = NotFound desc = could not find container \"88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511\": container with ID starting with 88cf9125bb428b25180988306b6764d49e5cd496f337c95661167a2da84dd511 not found: ID does not exist"
Sep 17 16:09:10.659989 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.659956    2584 scope.go:117] "RemoveContainer" containerID="5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf"
Sep 17 16:09:10.660158 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:09:10.660141    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf\": container with ID starting with 5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf not found: ID does not exist" containerID="5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf"
Sep 17 16:09:10.660196 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:10.660160    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf"} err="failed to get container status \"5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf\": rpc error: code = NotFound desc = could not find container \"5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf\": container with ID starting with 5bc721392407c2a22f04dbcd873aaf3c2e1630a0ec350e1e5fabd1c9dd6e79cf not found: ID does not exist"
Sep 17 16:09:12.370858 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:12.370829    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" path="/var/lib/kubelet/pods/954a3e80-2531-4339-bb8e-cace724d0a92/volumes"
Sep 17 16:09:17.530540 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:17.530496    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:09:27.531267 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:27.531224    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:09:37.530742 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:37.530699    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:09:47.531348 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:47.531306    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:09:49.180658 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:49.180595    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:49.272577 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:49.272545    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:09:57.366617 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:09:57.366586    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:10:04.879797 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:04.879756    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"]
Sep 17 16:10:04.880206 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:04.880017    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" containerID="cri-o://1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0" gracePeriod=30
Sep 17 16:10:04.880206 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:04.880051    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kube-rbac-proxy" containerID="cri-o://98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1" gracePeriod=30
Sep 17 16:10:04.997141 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:04.997110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:05.825572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:05.825538    2584 generic.go:356] "Generic (PLEG): container finished" podID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerID="98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1" exitCode=2
Sep 17 16:10:05.825749 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:05.825610    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" event={"ID":"43c32301-34fc-464c-a8f4-62593e3d41ef","Type":"ContainerDied","Data":"98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1"}
Sep 17 16:10:07.366425 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:07.366381    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 17 16:10:07.525891 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:07.525848    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 17 16:10:09.717700 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.717674    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:10:09.766322 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.766287    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c32301-34fc-464c-a8f4-62593e3d41ef-proxy-tls\") pod \"43c32301-34fc-464c-a8f4-62593e3d41ef\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") "
Sep 17 16:10:09.766467 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.766336    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-rbac-proxy-sar-config\") pod \"43c32301-34fc-464c-a8f4-62593e3d41ef\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") "
Sep 17 16:10:09.766467 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.766359    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c32301-34fc-464c-a8f4-62593e3d41ef-kserve-provision-location\") pod \"43c32301-34fc-464c-a8f4-62593e3d41ef\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") "
Sep 17 16:10:09.766467 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.766383    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-cabundle-cert\") pod \"43c32301-34fc-464c-a8f4-62593e3d41ef\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") "
Sep 17 16:10:09.766467 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.766415    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qp58\" (UniqueName: \"kubernetes.io/projected/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-api-access-4qp58\") pod \"43c32301-34fc-464c-a8f4-62593e3d41ef\" (UID: \"43c32301-34fc-464c-a8f4-62593e3d41ef\") "
Sep 17 16:10:09.766639 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.766618    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43c32301-34fc-464c-a8f4-62593e3d41ef-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43c32301-34fc-464c-a8f4-62593e3d41ef" (UID: "43c32301-34fc-464c-a8f4-62593e3d41ef"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:09.766801 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.766721    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "43c32301-34fc-464c-a8f4-62593e3d41ef" (UID: "43c32301-34fc-464c-a8f4-62593e3d41ef"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:09.766950 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.766918    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43c32301-34fc-464c-a8f4-62593e3d41ef" (UID: "43c32301-34fc-464c-a8f4-62593e3d41ef"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:09.768541 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.768516    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c32301-34fc-464c-a8f4-62593e3d41ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43c32301-34fc-464c-a8f4-62593e3d41ef" (UID: "43c32301-34fc-464c-a8f4-62593e3d41ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:10:09.768541 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.768518    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-api-access-4qp58" (OuterVolumeSpecName: "kube-api-access-4qp58") pod "43c32301-34fc-464c-a8f4-62593e3d41ef" (UID: "43c32301-34fc-464c-a8f4-62593e3d41ef"). InnerVolumeSpecName "kube-api-access-4qp58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:10:09.846334 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.846303    2584 generic.go:356] "Generic (PLEG): container finished" podID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerID="1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0" exitCode=0
Sep 17 16:10:09.846476 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.846346    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" event={"ID":"43c32301-34fc-464c-a8f4-62593e3d41ef","Type":"ContainerDied","Data":"1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0"}
Sep 17 16:10:09.846476 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.846361    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"
Sep 17 16:10:09.846476 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.846394    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g" event={"ID":"43c32301-34fc-464c-a8f4-62593e3d41ef","Type":"ContainerDied","Data":"933f55bcc10e81728ba0917eeec4850a813917475a3586825601a8e7ee7218ba"}
Sep 17 16:10:09.846476 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.846420    2584 scope.go:117] "RemoveContainer" containerID="98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1"
Sep 17 16:10:09.854316 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.854297    2584 scope.go:117] "RemoveContainer" containerID="1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0"
Sep 17 16:10:09.861187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.861171    2584 scope.go:117] "RemoveContainer" containerID="d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a"
Sep 17 16:10:09.866896 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.866875    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qp58\" (UniqueName: \"kubernetes.io/projected/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-api-access-4qp58\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.866995 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.866901    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43c32301-34fc-464c-a8f4-62593e3d41ef-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.866995 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.866917    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.866995 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.866928    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43c32301-34fc-464c-a8f4-62593e3d41ef-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.866995 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.866936    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/43c32301-34fc-464c-a8f4-62593e3d41ef-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.868395 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.868362    2584 scope.go:117] "RemoveContainer" containerID="98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1"
Sep 17 16:10:09.868733 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:10:09.868707    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1\": container with ID starting with 98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1 not found: ID does not exist" containerID="98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1"
Sep 17 16:10:09.868819 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.868744    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1"} err="failed to get container status \"98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1\": rpc error: code = NotFound desc = could not find container \"98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1\": container with ID starting with 98427f14291ef640ab049bc332b3175c478da7d058a3d165d443e344e59a26e1 not found: ID does not exist"
Sep 17 16:10:09.868819 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.868768    2584 scope.go:117] "RemoveContainer" containerID="1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0"
Sep 17 16:10:09.869133 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:10:09.869116    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0\": container with ID starting with 1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0 not found: ID does not exist" containerID="1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0"
Sep 17 16:10:09.869184 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.869138    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0"} err="failed to get container status \"1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0\": rpc error: code = NotFound desc = could not find container \"1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0\": container with ID starting with 1f930be7573b06c26eaacbecef47bf8ea5dc3961d2160922de8ce596e7820dd0 not found: ID does not exist"
Sep 17 16:10:09.869184 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.869154    2584 scope.go:117] "RemoveContainer" containerID="d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a"
Sep 17 16:10:09.869412 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:10:09.869397    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a\": container with ID starting with d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a not found: ID does not exist" containerID="d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a"
Sep 17 16:10:09.869466 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.869413    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a"} err="failed to get container status \"d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a\": rpc error: code = NotFound desc = could not find container \"d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a\": container with ID starting with d61e6f85028fcb9c2b9bbfb6d8b196daa02a745c528813a4ae508385ad59ed5a not found: ID does not exist"
Sep 17 16:10:09.869576 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.869560    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"]
Sep 17 16:10:09.873124 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:09.873105    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b9dfd44-4n79g"]
Sep 17 16:10:10.030537 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.030500    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:10.095719 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.095685    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"]
Sep 17 16:10:10.095982 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.095968    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container"
Sep 17 16:10:10.096031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.095984    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container"
Sep 17 16:10:10.096031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.095994    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent"
Sep 17 16:10:10.096031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.095999    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent"
Sep 17 16:10:10.096031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096007    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kube-rbac-proxy"
Sep 17 16:10:10.096031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096013    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kube-rbac-proxy"
Sep 17 16:10:10.096031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096021    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy"
Sep 17 16:10:10.096031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096026    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096036    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="storage-initializer"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096041    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="storage-initializer"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096049    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096054    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096066    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="storage-initializer"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096071    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="storage-initializer"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096108    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kserve-container"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096114    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kserve-container"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096121    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="kube-rbac-proxy"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096126    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="954a3e80-2531-4339-bb8e-cace724d0a92" containerName="agent"
Sep 17 16:10:10.096228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.096130    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" containerName="kube-rbac-proxy"
Sep 17 16:10:10.100436 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.100384    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.103404 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.103383    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 16:10:10.103523 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.103454    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:10:10.103587 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.103456    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 16:10:10.103587 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.103488    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:10:10.103723 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.103689    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:10:10.103829 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.103812    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:10:10.103923 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.103908    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z8dwv\""
Sep 17 16:10:10.105445 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.105426    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"]
Sep 17 16:10:10.108499 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.108481    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.110981 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.110960    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 16:10:10.111699 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.111680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"]
Sep 17 16:10:10.114480 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.114359    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:10.128487 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.128468    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"]
Sep 17 16:10:10.169427 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169390    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5jd\" (UniqueName: \"kubernetes.io/projected/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-api-access-mv5jd\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.169571 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169434    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d369865-8f7b-4716-be75-03e4f8e310bd-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.169571 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169455    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.169571 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169525    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d369865-8f7b-4716-be75-03e4f8e310bd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.169571 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169569    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.169754 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169586    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.169754 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169606    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.169754 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169624    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls48p\" (UniqueName: \"kubernetes.io/projected/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-api-access-ls48p\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.169754 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169702    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.169754 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.169738    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.185297 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.185269    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:10.249086 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.249058    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:10.271120 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271071    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.271120 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.271375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271136    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.271375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271165    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls48p\" (UniqueName: \"kubernetes.io/projected/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-api-access-ls48p\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.271375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271203    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.271524 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271405    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.271524 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271483    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5jd\" (UniqueName: \"kubernetes.io/projected/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-api-access-mv5jd\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.271624 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271530    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d369865-8f7b-4716-be75-03e4f8e310bd-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.271624 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271561    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.271624 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271588    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d369865-8f7b-4716-be75-03e4f8e310bd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.271624 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271597    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.272012 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271982    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.272093 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.271985    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.272349 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.272209    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d369865-8f7b-4716-be75-03e4f8e310bd-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.272349 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.272224    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.272349 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.272262    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.274385 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.274357    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.274502 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.274420    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d369865-8f7b-4716-be75-03e4f8e310bd-proxy-tls\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.280139 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.280119    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5jd\" (UniqueName: \"kubernetes.io/projected/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-api-access-mv5jd\") pod \"isvc-canary-stop-v2-predictor-6786bd6d5-bwncf\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.280226 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.280144    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls48p\" (UniqueName: \"kubernetes.io/projected/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-api-access-ls48p\") pod \"isvc-canary-stop-predictor-7f68f9d4d-8d72r\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.370658 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.370567    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43c32301-34fc-464c-a8f4-62593e3d41ef" path="/var/lib/kubelet/pods/43c32301-34fc-464c-a8f4-62593e3d41ef/volumes"
Sep 17 16:10:10.411349 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.411315    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:10.420404 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.420383    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:10.684354 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.684329    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"]
Sep 17 16:10:10.686949 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:10:10.686924    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2240e96e_9057_46c1_8cc8_f3b56c0dbe29.slice/crio-a914d8d3a2190acf59209e7f86ea2359b90df2c052c8b0159bbcdf94303a3d02 WatchSource:0}: Error finding container a914d8d3a2190acf59209e7f86ea2359b90df2c052c8b0159bbcdf94303a3d02: Status 404 returned error can't find the container with id a914d8d3a2190acf59209e7f86ea2359b90df2c052c8b0159bbcdf94303a3d02
Sep 17 16:10:10.691715 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.691687    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"]
Sep 17 16:10:10.694831 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:10:10.694805    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d369865_8f7b_4716_be75_03e4f8e310bd.slice/crio-e2fbeea09a4f6c263b383a9dbc63e2b1fd82af6a01a103ac6185841df097ac01 WatchSource:0}: Error finding container e2fbeea09a4f6c263b383a9dbc63e2b1fd82af6a01a103ac6185841df097ac01: Status 404 returned error can't find the container with id e2fbeea09a4f6c263b383a9dbc63e2b1fd82af6a01a103ac6185841df097ac01
Sep 17 16:10:10.853307 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.853264    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" event={"ID":"8d369865-8f7b-4716-be75-03e4f8e310bd","Type":"ContainerStarted","Data":"20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7"}
Sep 17 16:10:10.853735 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.853372    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" event={"ID":"8d369865-8f7b-4716-be75-03e4f8e310bd","Type":"ContainerStarted","Data":"e2fbeea09a4f6c263b383a9dbc63e2b1fd82af6a01a103ac6185841df097ac01"}
Sep 17 16:10:10.855105 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.855073    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" event={"ID":"2240e96e-9057-46c1-8cc8-f3b56c0dbe29","Type":"ContainerStarted","Data":"03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c"}
Sep 17 16:10:10.855213 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:10.855149    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" event={"ID":"2240e96e-9057-46c1-8cc8-f3b56c0dbe29","Type":"ContainerStarted","Data":"a914d8d3a2190acf59209e7f86ea2359b90df2c052c8b0159bbcdf94303a3d02"}
Sep 17 16:10:14.870121 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:14.870031    2584 generic.go:356] "Generic (PLEG): container finished" podID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerID="20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7" exitCode=0
Sep 17 16:10:14.870121 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:14.870102    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" event={"ID":"8d369865-8f7b-4716-be75-03e4f8e310bd","Type":"ContainerDied","Data":"20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7"}
Sep 17 16:10:14.871730 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:14.871712    2584 generic.go:356] "Generic (PLEG): container finished" podID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerID="03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c" exitCode=0
Sep 17 16:10:14.871824 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:14.871798    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" event={"ID":"2240e96e-9057-46c1-8cc8-f3b56c0dbe29","Type":"ContainerDied","Data":"03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c"}
Sep 17 16:10:15.877964 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.877919    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" event={"ID":"8d369865-8f7b-4716-be75-03e4f8e310bd","Type":"ContainerStarted","Data":"27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019"}
Sep 17 16:10:15.878405 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.878034    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" event={"ID":"8d369865-8f7b-4716-be75-03e4f8e310bd","Type":"ContainerStarted","Data":"6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a"}
Sep 17 16:10:15.878405 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.878249    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:15.878599 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.878405    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:15.879583 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.879559    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:10:15.880158 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.880132    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" event={"ID":"2240e96e-9057-46c1-8cc8-f3b56c0dbe29","Type":"ContainerStarted","Data":"9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35"}
Sep 17 16:10:15.880246 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.880174    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" event={"ID":"2240e96e-9057-46c1-8cc8-f3b56c0dbe29","Type":"ContainerStarted","Data":"544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e"}
Sep 17 16:10:15.880366 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.880351    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:15.900750 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.900701    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podStartSLOduration=2.096562684 podStartE2EDuration="5.900685484s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8041228s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:15.900057168 +0000 UTC m=+974.126726473" watchObservedRunningTime="2026-09-17 16:10:15.900685484 +0000 UTC m=+974.127354836"
Sep 17 16:10:15.924026 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:15.923955    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podStartSLOduration=2.096899268 podStartE2EDuration="5.923898119s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.826998851s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:15.922874583 +0000 UTC m=+974.149543896" watchObservedRunningTime="2026-09-17 16:10:15.923898119 +0000 UTC m=+974.150567428"
Sep 17 16:10:16.884203 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:16.884175    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:16.884651 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:16.884215    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:10:16.885610 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:16.885586    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:10:17.888053 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:17.888006    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:10:19.966042 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:19.966011    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"]
Sep 17 16:10:19.969577 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:19.969554    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:19.970854 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:19.970826    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:19.972317 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:19.972301    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-4eea9-kube-rbac-proxy-sar-config\""
Sep 17 16:10:19.972530 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:19.972513    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-4eea9-serving-cert\""
Sep 17 16:10:19.978896 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:19.978872    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"]
Sep 17 16:10:20.050732 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.050692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-openshift-service-ca-bundle\") pod \"model-chainer-raw-4eea9-55c59cd6bf-h6tng\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") " pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:20.050732 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.050736    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls\") pod \"model-chainer-raw-4eea9-55c59cd6bf-h6tng\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") " pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:20.151498 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.151464    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-openshift-service-ca-bundle\") pod \"model-chainer-raw-4eea9-55c59cd6bf-h6tng\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") " pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:20.151687 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.151502    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls\") pod \"model-chainer-raw-4eea9-55c59cd6bf-h6tng\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") " pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:20.151687 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:10:20.151638    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-raw-4eea9-serving-cert: secret "model-chainer-raw-4eea9-serving-cert" not found
Sep 17 16:10:20.151825 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:10:20.151699    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls podName:5ba33d59-0ddd-446a-83c2-8e370bc1d3ef nodeName:}" failed. No retries permitted until 2026-09-17 16:10:20.651684228 +0000 UTC m=+978.878353493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls") pod "model-chainer-raw-4eea9-55c59cd6bf-h6tng" (UID: "5ba33d59-0ddd-446a-83c2-8e370bc1d3ef") : secret "model-chainer-raw-4eea9-serving-cert" not found
Sep 17 16:10:20.152211 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.152189    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-openshift-service-ca-bundle\") pod \"model-chainer-raw-4eea9-55c59cd6bf-h6tng\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") " pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:20.243854 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.243756    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:20.656542 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.656492    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls\") pod \"model-chainer-raw-4eea9-55c59cd6bf-h6tng\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") " pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:20.658920 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.658893    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls\") pod \"model-chainer-raw-4eea9-55c59cd6bf-h6tng\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") " pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:20.880250 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:20.880210    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:21.157511 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:21.157482    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"]
Sep 17 16:10:21.163587 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:10:21.163559    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba33d59_0ddd_446a_83c2_8e370bc1d3ef.slice/crio-4770706aa528a0162bacc711564fc842873384312d1c526c3dd625dceb08a402 WatchSource:0}: Error finding container 4770706aa528a0162bacc711564fc842873384312d1c526c3dd625dceb08a402: Status 404 returned error can't find the container with id 4770706aa528a0162bacc711564fc842873384312d1c526c3dd625dceb08a402
Sep 17 16:10:21.165492 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:21.165469    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:10:21.889300 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:21.889272    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:10:21.889807 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:21.889763    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:10:21.902453 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:21.902425    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" event={"ID":"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef","Type":"ContainerStarted","Data":"4770706aa528a0162bacc711564fc842873384312d1c526c3dd625dceb08a402"}
Sep 17 16:10:22.893459 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:22.893432    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:10:22.894399 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:22.894041    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:10:23.911563 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:23.911517    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" event={"ID":"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef","Type":"ContainerStarted","Data":"a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5"}
Sep 17 16:10:23.911947 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:23.911641    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:23.927531 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:23.927476    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" podStartSLOduration=2.596333948 podStartE2EDuration="4.927458171s" totalImagesPullingTime="2.331124223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:23.927226908 +0000 UTC m=+982.153896198" watchObservedRunningTime="2026-09-17 16:10:23.927458171 +0000 UTC m=+982.154127459"
Sep 17 16:10:29.922391 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:29.922290    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:30.056491 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.056461    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"]
Sep 17 16:10:30.056677 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.056636    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9" containerID="cri-o://a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5" gracePeriod=30
Sep 17 16:10:30.080014 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.079986    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:30.201146 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.201061    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:30.289557 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.289518    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:30.384879 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.384831    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:30.462477 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.462391    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:30.787285 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.787202    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:30.904549 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:30.904521    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:10:31.890380 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:31.890344    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:10:32.894380 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:32.894344    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:10:34.920660 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:34.920619    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:39.919994 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:39.919954    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:41.889915 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:41.889872    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:10:42.894402 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:42.894358    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:10:44.919703 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:44.919656    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:44.920150 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:44.919807    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:10:49.920280 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:49.920237    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:51.890186 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:51.890145    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:10:52.893975 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:52.893936    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:10:54.920831 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:54.920791    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:10:59.920988 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:10:59.920941    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:11:00.195188 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:00.195162    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:11:00.348347 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:00.348308    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-openshift-service-ca-bundle\") pod \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") "
Sep 17 16:11:00.348536 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:00.348361    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls\") pod \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\" (UID: \"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef\") "
Sep 17 16:11:00.348686 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:00.348661    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" (UID: "5ba33d59-0ddd-446a-83c2-8e370bc1d3ef"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:00.350383 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:00.350358    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" (UID: "5ba33d59-0ddd-446a-83c2-8e370bc1d3ef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:11:00.449127 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:00.449043    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-openshift-service-ca-bundle\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:00.449127 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:00.449073    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:01.053604 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.053573    2584 generic.go:356] "Generic (PLEG): container finished" podID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerID="a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5" exitCode=0
Sep 17 16:11:01.054136 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.053626    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"
Sep 17 16:11:01.054136 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.053626    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" event={"ID":"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef","Type":"ContainerDied","Data":"a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5"}
Sep 17 16:11:01.054136 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.053679    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng" event={"ID":"5ba33d59-0ddd-446a-83c2-8e370bc1d3ef","Type":"ContainerDied","Data":"4770706aa528a0162bacc711564fc842873384312d1c526c3dd625dceb08a402"}
Sep 17 16:11:01.054136 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.053702    2584 scope.go:117] "RemoveContainer" containerID="a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5"
Sep 17 16:11:01.061658 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.061483    2584 scope.go:117] "RemoveContainer" containerID="a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5"
Sep 17 16:11:01.061812 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:11:01.061774    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5\": container with ID starting with a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5 not found: ID does not exist" containerID="a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5"
Sep 17 16:11:01.061859 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.061821    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5"} err="failed to get container status \"a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5\": rpc error: code = NotFound desc = could not find container \"a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5\": container with ID starting with a86a47fc644d78c495f0ee21900daef27f9fff363a5afcffa2970744c05de6f5 not found: ID does not exist"
Sep 17 16:11:01.069688 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.069668    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"]
Sep 17 16:11:01.075823 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.075802    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-4eea9-55c59cd6bf-h6tng"]
Sep 17 16:11:01.889719 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:01.889682    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:11:02.369985 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:02.369958    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" path="/var/lib/kubelet/pods/5ba33d59-0ddd-446a-83c2-8e370bc1d3ef/volumes"
Sep 17 16:11:02.894152 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:02.894112    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:11:11.890542 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:11.890503    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:11:12.894487 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:12.894444    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:11:21.890278 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:21.890234    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:11:22.894238 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:22.894197    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:11:28.366123 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:28.366081    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 17 16:11:31.891100 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:31.891067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:11:38.370424 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:38.370395    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:11:40.048396 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.048362    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"]
Sep 17 16:11:40.048889 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.048644    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container" containerID="cri-o://544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e" gracePeriod=30
Sep 17 16:11:40.048889 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.048711    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kube-rbac-proxy" containerID="cri-o://9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35" gracePeriod=30
Sep 17 16:11:40.073223 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.073189    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:40.080371 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.080346    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"]
Sep 17 16:11:40.080631 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.080606    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" containerID="cri-o://6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a" gracePeriod=30
Sep 17 16:11:40.080682 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.080619    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kube-rbac-proxy" containerID="cri-o://27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019" gracePeriod=30
Sep 17 16:11:40.197468 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.197441    2584 generic.go:356] "Generic (PLEG): container finished" podID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerID="27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019" exitCode=2
Sep 17 16:11:40.197575 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.197514    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" event={"ID":"8d369865-8f7b-4716-be75-03e4f8e310bd","Type":"ContainerDied","Data":"27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019"}
Sep 17 16:11:40.199525 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.199506    2584 generic.go:356] "Generic (PLEG): container finished" podID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerID="9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35" exitCode=2
Sep 17 16:11:40.199624 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:40.199559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" event={"ID":"2240e96e-9057-46c1-8cc8-f3b56c0dbe29","Type":"ContainerDied","Data":"9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35"}
Sep 17 16:11:41.884719 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:41.884671    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 17 16:11:41.890046 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:41.890008    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 17 16:11:42.320368 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:42.320326    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:42.888765 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:42.888726    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 17 16:11:44.999366 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:44.999342    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:11:45.068085 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068055    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ls48p\" (UniqueName: \"kubernetes.io/projected/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-api-access-ls48p\") pod \"8d369865-8f7b-4716-be75-03e4f8e310bd\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") "
Sep 17 16:11:45.068211 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068103    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-rbac-proxy-sar-config\") pod \"8d369865-8f7b-4716-be75-03e4f8e310bd\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") "
Sep 17 16:11:45.068211 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068139    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d369865-8f7b-4716-be75-03e4f8e310bd-kserve-provision-location\") pod \"8d369865-8f7b-4716-be75-03e4f8e310bd\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") "
Sep 17 16:11:45.068211 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068160    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-cabundle-cert\") pod \"8d369865-8f7b-4716-be75-03e4f8e310bd\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") "
Sep 17 16:11:45.068384 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068360    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d369865-8f7b-4716-be75-03e4f8e310bd-proxy-tls\") pod \"8d369865-8f7b-4716-be75-03e4f8e310bd\" (UID: \"8d369865-8f7b-4716-be75-03e4f8e310bd\") "
Sep 17 16:11:45.068500 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068467    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d369865-8f7b-4716-be75-03e4f8e310bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8d369865-8f7b-4716-be75-03e4f8e310bd" (UID: "8d369865-8f7b-4716-be75-03e4f8e310bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:45.068615 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068518    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8d369865-8f7b-4716-be75-03e4f8e310bd" (UID: "8d369865-8f7b-4716-be75-03e4f8e310bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:45.068615 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068521    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8d369865-8f7b-4716-be75-03e4f8e310bd" (UID: "8d369865-8f7b-4716-be75-03e4f8e310bd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:45.068722 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068645    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.068722 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068664    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8d369865-8f7b-4716-be75-03e4f8e310bd-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.068722 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.068678    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8d369865-8f7b-4716-be75-03e4f8e310bd-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.070932 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.070909    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-api-access-ls48p" (OuterVolumeSpecName: "kube-api-access-ls48p") pod "8d369865-8f7b-4716-be75-03e4f8e310bd" (UID: "8d369865-8f7b-4716-be75-03e4f8e310bd"). InnerVolumeSpecName "kube-api-access-ls48p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:11:45.071020 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.070954    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d369865-8f7b-4716-be75-03e4f8e310bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8d369865-8f7b-4716-be75-03e4f8e310bd" (UID: "8d369865-8f7b-4716-be75-03e4f8e310bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:11:45.090797 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.090761    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:11:45.169546 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.169506    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kserve-provision-location\") pod \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") "
Sep 17 16:11:45.169733 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.169557    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-rbac-proxy-sar-config\") pod \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") "
Sep 17 16:11:45.169733 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.169602    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv5jd\" (UniqueName: \"kubernetes.io/projected/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-api-access-mv5jd\") pod \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") "
Sep 17 16:11:45.169733 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.169637    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-cabundle-cert\") pod \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") "
Sep 17 16:11:45.169733 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.169658    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-proxy-tls\") pod \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\" (UID: \"2240e96e-9057-46c1-8cc8-f3b56c0dbe29\") "
Sep 17 16:11:45.169981 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.169843    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8d369865-8f7b-4716-be75-03e4f8e310bd-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.169981 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.169860    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls48p\" (UniqueName: \"kubernetes.io/projected/8d369865-8f7b-4716-be75-03e4f8e310bd-kube-api-access-ls48p\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.170068 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.169989    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2240e96e-9057-46c1-8cc8-f3b56c0dbe29" (UID: "2240e96e-9057-46c1-8cc8-f3b56c0dbe29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:45.170068 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.170004    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2240e96e-9057-46c1-8cc8-f3b56c0dbe29" (UID: "2240e96e-9057-46c1-8cc8-f3b56c0dbe29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:45.170068 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.170049    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2240e96e-9057-46c1-8cc8-f3b56c0dbe29" (UID: "2240e96e-9057-46c1-8cc8-f3b56c0dbe29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:45.171683 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.171655    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2240e96e-9057-46c1-8cc8-f3b56c0dbe29" (UID: "2240e96e-9057-46c1-8cc8-f3b56c0dbe29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:11:45.171774 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.171698    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-api-access-mv5jd" (OuterVolumeSpecName: "kube-api-access-mv5jd") pod "2240e96e-9057-46c1-8cc8-f3b56c0dbe29" (UID: "2240e96e-9057-46c1-8cc8-f3b56c0dbe29"). InnerVolumeSpecName "kube-api-access-mv5jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:11:45.220831 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.220800    2584 generic.go:356] "Generic (PLEG): container finished" podID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerID="6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a" exitCode=0
Sep 17 16:11:45.221013 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.220875    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"
Sep 17 16:11:45.221013 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.220880    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" event={"ID":"8d369865-8f7b-4716-be75-03e4f8e310bd","Type":"ContainerDied","Data":"6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a"}
Sep 17 16:11:45.221013 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.220947    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r" event={"ID":"8d369865-8f7b-4716-be75-03e4f8e310bd","Type":"ContainerDied","Data":"e2fbeea09a4f6c263b383a9dbc63e2b1fd82af6a01a103ac6185841df097ac01"}
Sep 17 16:11:45.221013 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.220985    2584 scope.go:117] "RemoveContainer" containerID="27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019"
Sep 17 16:11:45.222930 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.222909    2584 generic.go:356] "Generic (PLEG): container finished" podID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerID="544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e" exitCode=0
Sep 17 16:11:45.223066 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.222941    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" event={"ID":"2240e96e-9057-46c1-8cc8-f3b56c0dbe29","Type":"ContainerDied","Data":"544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e"}
Sep 17 16:11:45.223066 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.222970    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"
Sep 17 16:11:45.223066 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.222981    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf" event={"ID":"2240e96e-9057-46c1-8cc8-f3b56c0dbe29","Type":"ContainerDied","Data":"a914d8d3a2190acf59209e7f86ea2359b90df2c052c8b0159bbcdf94303a3d02"}
Sep 17 16:11:45.230215 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.229967    2584 scope.go:117] "RemoveContainer" containerID="6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a"
Sep 17 16:11:45.237412 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.237396    2584 scope.go:117] "RemoveContainer" containerID="20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7"
Sep 17 16:11:45.244224 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.244196    2584 scope.go:117] "RemoveContainer" containerID="27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019"
Sep 17 16:11:45.244658 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:11:45.244594    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019\": container with ID starting with 27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019 not found: ID does not exist" containerID="27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019"
Sep 17 16:11:45.244658 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.244628    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019"} err="failed to get container status \"27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019\": rpc error: code = NotFound desc = could not find container \"27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019\": container with ID starting with 27cc1656b40e1888a9eaa0c38cf0f8865f6038f6c5aeca4b50bb9ab47478d019 not found: ID does not exist"
Sep 17 16:11:45.244658 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.244650    2584 scope.go:117] "RemoveContainer" containerID="6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a"
Sep 17 16:11:45.245170 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:11:45.245145    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a\": container with ID starting with 6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a not found: ID does not exist" containerID="6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a"
Sep 17 16:11:45.245262 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.245178    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a"} err="failed to get container status \"6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a\": rpc error: code = NotFound desc = could not find container \"6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a\": container with ID starting with 6f7b33145cd0f72433a3869aabee7971a788f803bf609268e009433dcbf8526a not found: ID does not exist"
Sep 17 16:11:45.245262 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.245198    2584 scope.go:117] "RemoveContainer" containerID="20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7"
Sep 17 16:11:45.245533 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:11:45.245516    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7\": container with ID starting with 20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7 not found: ID does not exist" containerID="20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7"
Sep 17 16:11:45.245619 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.245537    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7"} err="failed to get container status \"20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7\": rpc error: code = NotFound desc = could not find container \"20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7\": container with ID starting with 20c27d22c3a739848fa496aab46a1ba63931d8ed4becc51021dc543f619340e7 not found: ID does not exist"
Sep 17 16:11:45.245619 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.245550    2584 scope.go:117] "RemoveContainer" containerID="9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35"
Sep 17 16:11:45.246329 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.246308    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"]
Sep 17 16:11:45.250676 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.250656    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f68f9d4d-8d72r"]
Sep 17 16:11:45.257301 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.257282    2584 scope.go:117] "RemoveContainer" containerID="544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e"
Sep 17 16:11:45.262398 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.262376    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"]
Sep 17 16:11:45.264920 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.264903    2584 scope.go:117] "RemoveContainer" containerID="03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c"
Sep 17 16:11:45.266903 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.266881    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6786bd6d5-bwncf"]
Sep 17 16:11:45.270448 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.270427    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv5jd\" (UniqueName: \"kubernetes.io/projected/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-api-access-mv5jd\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.270528 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.270450    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.270528 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.270460    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.270528 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.270468    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.270528 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.270476    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2240e96e-9057-46c1-8cc8-f3b56c0dbe29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:11:45.272180 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.272163    2584 scope.go:117] "RemoveContainer" containerID="9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35"
Sep 17 16:11:45.272437 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:11:45.272418    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35\": container with ID starting with 9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35 not found: ID does not exist" containerID="9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35"
Sep 17 16:11:45.272484 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.272445    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35"} err="failed to get container status \"9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35\": rpc error: code = NotFound desc = could not find container \"9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35\": container with ID starting with 9370638df6bdaac923454f201845f5db4774f4a8a1e699e436e07c0c2daffa35 not found: ID does not exist"
Sep 17 16:11:45.272484 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.272461    2584 scope.go:117] "RemoveContainer" containerID="544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e"
Sep 17 16:11:45.272710 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:11:45.272694    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e\": container with ID starting with 544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e not found: ID does not exist" containerID="544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e"
Sep 17 16:11:45.272765 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.272713    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e"} err="failed to get container status \"544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e\": rpc error: code = NotFound desc = could not find container \"544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e\": container with ID starting with 544a2ec178f3909c82764d2a92bc79f5fefab57c29c6a72508f7668ca85ea29e not found: ID does not exist"
Sep 17 16:11:45.272765 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.272727    2584 scope.go:117] "RemoveContainer" containerID="03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c"
Sep 17 16:11:45.272978 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:11:45.272963    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c\": container with ID starting with 03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c not found: ID does not exist" containerID="03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c"
Sep 17 16:11:45.273014 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:45.272982    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c"} err="failed to get container status \"03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c\": rpc error: code = NotFound desc = could not find container \"03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c\": container with ID starting with 03c57ebec4bef94ed1d7a61adf804f25393eb65a640ea442c60aba3db63aa79c not found: ID does not exist"
Sep 17 16:11:46.370586 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:46.370553    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" path="/var/lib/kubelet/pods/2240e96e-9057-46c1-8cc8-f3b56c0dbe29/volumes"
Sep 17 16:11:46.371117 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:46.371102    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" path="/var/lib/kubelet/pods/8d369865-8f7b-4716-be75-03e4f8e310bd/volumes"
Sep 17 16:11:50.293724 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:50.293687    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:11:50.401341 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:11:50.401281    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:12.300211 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:12.300173    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:20.318374 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318341    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"]
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318597    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="storage-initializer"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318609    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="storage-initializer"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318617    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318623    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318635    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kube-rbac-proxy"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318642    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kube-rbac-proxy"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318652    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kube-rbac-proxy"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318658    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kube-rbac-proxy"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318667    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="storage-initializer"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318672    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="storage-initializer"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318677    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318682    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318689    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318694    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318736    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kube-rbac-proxy"
Sep 17 16:12:20.318739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318745    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba33d59-0ddd-446a-83c2-8e370bc1d3ef" containerName="model-chainer-raw-4eea9"
Sep 17 16:12:20.319238 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318752    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2240e96e-9057-46c1-8cc8-f3b56c0dbe29" containerName="kserve-container"
Sep 17 16:12:20.319238 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318762    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kube-rbac-proxy"
Sep 17 16:12:20.319238 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.318767    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d369865-8f7b-4716-be75-03e4f8e310bd" containerName="kserve-container"
Sep 17 16:12:20.323028 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.323007    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:20.326121 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.326097    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z8dwv\""
Sep 17 16:12:20.326121 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.326110    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-aa09f-kube-rbac-proxy-sar-config\""
Sep 17 16:12:20.326121 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.326104    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-raw-hpa-aa09f-serving-cert\""
Sep 17 16:12:20.326388 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.326115    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:12:20.332708 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.332680    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"]
Sep 17 16:12:20.334576 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.334546    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:20.426229 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.426190    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e72bd4-98a9-4e48-8394-1fef1b02f73d-proxy-tls\") pod \"model-chainer-raw-hpa-aa09f-686678f68d-ndpqd\" (UID: \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:20.426408 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.426297    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e72bd4-98a9-4e48-8394-1fef1b02f73d-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-aa09f-686678f68d-ndpqd\" (UID: \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:20.431739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.431701    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:20.526874 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.526838    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e72bd4-98a9-4e48-8394-1fef1b02f73d-proxy-tls\") pod \"model-chainer-raw-hpa-aa09f-686678f68d-ndpqd\" (UID: \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:20.527043 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.526885    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e72bd4-98a9-4e48-8394-1fef1b02f73d-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-aa09f-686678f68d-ndpqd\" (UID: \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:20.527455 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.527437    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e72bd4-98a9-4e48-8394-1fef1b02f73d-openshift-service-ca-bundle\") pod \"model-chainer-raw-hpa-aa09f-686678f68d-ndpqd\" (UID: \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:20.529385 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.529360    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e72bd4-98a9-4e48-8394-1fef1b02f73d-proxy-tls\") pod \"model-chainer-raw-hpa-aa09f-686678f68d-ndpqd\" (UID: \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\") " pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:20.634475 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.634386    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:20.905932 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:20.905795    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"]
Sep 17 16:12:20.908552 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:12:20.908529    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e72bd4_98a9_4e48_8394_1fef1b02f73d.slice/crio-32dbc49856a617d0b6e36caf584c7bb46f8203d1fb4a86e76fde9c6c9bb7a363 WatchSource:0}: Error finding container 32dbc49856a617d0b6e36caf584c7bb46f8203d1fb4a86e76fde9c6c9bb7a363: Status 404 returned error can't find the container with id 32dbc49856a617d0b6e36caf584c7bb46f8203d1fb4a86e76fde9c6c9bb7a363
Sep 17 16:12:21.350231 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:21.350196    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" event={"ID":"94e72bd4-98a9-4e48-8394-1fef1b02f73d","Type":"ContainerStarted","Data":"fba045dc3acc6e570ff56634c7dc3ec601b38952133b5c56e4cdb51f5b1152c8"}
Sep 17 16:12:21.350589 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:21.350246    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" event={"ID":"94e72bd4-98a9-4e48-8394-1fef1b02f73d","Type":"ContainerStarted","Data":"32dbc49856a617d0b6e36caf584c7bb46f8203d1fb4a86e76fde9c6c9bb7a363"}
Sep 17 16:12:21.350589 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:21.350311    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:27.359572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:27.359542    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:27.376577 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:27.376513    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" podStartSLOduration=7.376495734 podStartE2EDuration="7.376495734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:12:21.371643741 +0000 UTC m=+1099.598313049" watchObservedRunningTime="2026-09-17 16:12:27.376495734 +0000 UTC m=+1105.603165023"
Sep 17 16:12:30.374034 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.373998    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"]
Sep 17 16:12:30.374445 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.374193    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f" containerID="cri-o://fba045dc3acc6e570ff56634c7dc3ec601b38952133b5c56e4cdb51f5b1152c8" gracePeriod=30
Sep 17 16:12:30.382366 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.382333    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:30.499348 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.499316    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:30.538175 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.538147    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"]
Sep 17 16:12:30.543070 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.543041    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:30.546868 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.546843    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-2b3c5-predictor-serving-cert\""
Sep 17 16:12:30.548092 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.548071    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-2b3c5-kube-rbac-proxy-sar-config\""
Sep 17 16:12:30.548092 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.548092    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:12:30.559762 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.559695    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"]
Sep 17 16:12:30.606416 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.606381    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:30.702981 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.702890    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:30.705899 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.705878    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b774l\" (UniqueName: \"kubernetes.io/projected/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-api-access-b774l\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:30.705974 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.705924    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:30.705974 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.705967    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:30.806876 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.806840    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:30.807014 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.806889    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:30.807014 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.806932    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b774l\" (UniqueName: \"kubernetes.io/projected/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-api-access-b774l\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:30.807014 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:12:30.807003    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-serving-cert: secret "message-dumper-raw-2b3c5-predictor-serving-cert" not found
Sep 17 16:12:30.807148 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:12:30.807075    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls podName:c6ca8f35-3a5c-4d6a-b87c-841d7ec20424 nodeName:}" failed. No retries permitted until 2026-09-17 16:12:31.307053434 +0000 UTC m=+1109.533722714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls") pod "message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" (UID: "c6ca8f35-3a5c-4d6a-b87c-841d7ec20424") : secret "message-dumper-raw-2b3c5-predictor-serving-cert" not found
Sep 17 16:12:30.807602 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.807580    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:30.815554 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:30.815524    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b774l\" (UniqueName: \"kubernetes.io/projected/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-api-access-b774l\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:31.180837 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:31.180802    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:31.310939 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:31.310890    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:31.313325 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:31.313302    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls\") pod \"message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") " pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:31.455007 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:31.454911    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:31.722700 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:31.722631    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"]
Sep 17 16:12:31.725473 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:12:31.725447    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ca8f35_3a5c_4d6a_b87c_841d7ec20424.slice/crio-c44f3e14bf4938e78f01ca92d8ac871efca0a63ccd474a413336735665a2df1e WatchSource:0}: Error finding container c44f3e14bf4938e78f01ca92d8ac871efca0a63ccd474a413336735665a2df1e: Status 404 returned error can't find the container with id c44f3e14bf4938e78f01ca92d8ac871efca0a63ccd474a413336735665a2df1e
Sep 17 16:12:32.358311 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:32.358269    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:32.389133 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:32.389099    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" event={"ID":"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424","Type":"ContainerStarted","Data":"c44f3e14bf4938e78f01ca92d8ac871efca0a63ccd474a413336735665a2df1e"}
Sep 17 16:12:33.394848 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:33.394812    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" event={"ID":"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424","Type":"ContainerStarted","Data":"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec"}
Sep 17 16:12:33.395255 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:33.394874    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" event={"ID":"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424","Type":"ContainerStarted","Data":"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161"}
Sep 17 16:12:33.395255 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:33.395000    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:33.412125 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:33.412072    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" podStartSLOduration=2.16013405 podStartE2EDuration="3.412042065s" totalImagesPullingTime="1.251908015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:12:33.411853384 +0000 UTC m=+1111.638522696" watchObservedRunningTime="2026-09-17 16:12:33.412042065 +0000 UTC m=+1111.638711350"
Sep 17 16:12:34.398241 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:34.398208    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:34.399828 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:34.399797    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:36.099361 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:36.099324    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:37.357556 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:37.357506    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:41.409898 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:41.409867    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:12:42.357714 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:42.357677    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:42.357899 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:42.357772    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:12:47.357595 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:47.357550    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:50.617735 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.617696    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:50.656127 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.656095    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"]
Sep 17 16:12:50.659750 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.659728    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.662631 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.662607    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-2b3c5-predictor-serving-cert\""
Sep 17 16:12:50.662760 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.662689    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:12:50.662760 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.662690    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:12:50.662906 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.662803    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-2b3c5-kube-rbac-proxy-sar-config\""
Sep 17 16:12:50.672891 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.672866    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"]
Sep 17 16:12:50.752414 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.752376    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.752584 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.752429    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-cabundle-cert\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.752584 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.752505    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.752584 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.752544    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kserve-provision-location\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.752584 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.752572    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d5zp\" (UniqueName: \"kubernetes.io/projected/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-api-access-7d5zp\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.853187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.853151    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d5zp\" (UniqueName: \"kubernetes.io/projected/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-api-access-7d5zp\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.853375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.853220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.853375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.853265    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-cabundle-cert\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.853375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.853285    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.853375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.853301    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kserve-provision-location\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.853574 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:12:50.853411    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-serving-cert: secret "isvc-logger-raw-2b3c5-predictor-serving-cert" not found
Sep 17 16:12:50.853574 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:12:50.853479    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls podName:52f5f061-e8f6-45cd-ba4d-26a0f0ddc829 nodeName:}" failed. No retries permitted until 2026-09-17 16:12:51.353460292 +0000 UTC m=+1129.580129563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls") pod "isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" (UID: "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829") : secret "isvc-logger-raw-2b3c5-predictor-serving-cert" not found
Sep 17 16:12:50.853690 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.853663    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kserve-provision-location\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.854000 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.853978    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.854101 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.854047    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-cabundle-cert\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:50.864143 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:50.864120    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d5zp\" (UniqueName: \"kubernetes.io/projected/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-api-access-7d5zp\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:51.042077 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:51.041982    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:12:51.357172 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:51.357141    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:51.359468 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:51.359441    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls\") pod \"isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:51.569955 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:51.569899    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:51.844547 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:51.844522    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"]
Sep 17 16:12:51.846737 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:12:51.846704    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f5f061_e8f6_45cd_ba4d_26a0f0ddc829.slice/crio-6cd12203c84e0c1c765f222d76209e7ce374cbcca1e095023180a1d91ed57709 WatchSource:0}: Error finding container 6cd12203c84e0c1c765f222d76209e7ce374cbcca1e095023180a1d91ed57709: Status 404 returned error can't find the container with id 6cd12203c84e0c1c765f222d76209e7ce374cbcca1e095023180a1d91ed57709
Sep 17 16:12:52.357930 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:52.357895    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:52.467226 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:52.467186    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerStarted","Data":"3b2875f4e800c09e843ab7531527b30cb3888525eb7490fe621ab4b659ab3bc8"}
Sep 17 16:12:52.467390 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:52.467251    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerStarted","Data":"6cd12203c84e0c1c765f222d76209e7ce374cbcca1e095023180a1d91ed57709"}
Sep 17 16:12:56.481323 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:56.481289    2584 generic.go:356] "Generic (PLEG): container finished" podID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerID="3b2875f4e800c09e843ab7531527b30cb3888525eb7490fe621ab4b659ab3bc8" exitCode=0
Sep 17 16:12:56.481692 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:56.481363    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerDied","Data":"3b2875f4e800c09e843ab7531527b30cb3888525eb7490fe621ab4b659ab3bc8"}
Sep 17 16:12:57.357709 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:57.357673    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:57.487608 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:57.487568    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerStarted","Data":"3b39152cc6414d37ad7a3058190fee372983cbddd8771d59d0c6ef1f361e01ab"}
Sep 17 16:12:57.488003 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:57.487640    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerStarted","Data":"27c1d2609694247067f4ac60179ac7b41c22c69217fdbdba03e13fe9a9136a6e"}
Sep 17 16:12:57.488003 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:57.487667    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerStarted","Data":"caf183681ff2c1674e9adf217f2cb8217177706e0fbdaed05bbe68befa95e75f"}
Sep 17 16:12:57.488003 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:57.487963    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:57.488119 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:57.488095    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:57.489227 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:57.489204    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:12:57.513072 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:57.513021    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podStartSLOduration=3.801503196 podStartE2EDuration="7.51300599s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.711502794s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:12:57.510946997 +0000 UTC m=+1135.737616300" watchObservedRunningTime="2026-09-17 16:12:57.51300599 +0000 UTC m=+1135.739675278"
Sep 17 16:12:58.495152 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:58.494336    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:12:58.495533 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:58.495202    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:12:58.495829 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:58.495777    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:12:59.496174 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:59.496141    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:12:59.496629 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:12:59.496516    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:13:00.501107 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.501077    2584 generic.go:356] "Generic (PLEG): container finished" podID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerID="fba045dc3acc6e570ff56634c7dc3ec601b38952133b5c56e4cdb51f5b1152c8" exitCode=0
Sep 17 16:13:00.501413 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.501147    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" event={"ID":"94e72bd4-98a9-4e48-8394-1fef1b02f73d","Type":"ContainerDied","Data":"fba045dc3acc6e570ff56634c7dc3ec601b38952133b5c56e4cdb51f5b1152c8"}
Sep 17 16:13:00.501413 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.501203    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd" event={"ID":"94e72bd4-98a9-4e48-8394-1fef1b02f73d","Type":"ContainerDied","Data":"32dbc49856a617d0b6e36caf584c7bb46f8203d1fb4a86e76fde9c6c9bb7a363"}
Sep 17 16:13:00.501413 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.501220    2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32dbc49856a617d0b6e36caf584c7bb46f8203d1fb4a86e76fde9c6c9bb7a363"
Sep 17 16:13:00.511649 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.511632    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:13:00.619235 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.619199    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e72bd4-98a9-4e48-8394-1fef1b02f73d-openshift-service-ca-bundle\") pod \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\" (UID: \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\") "
Sep 17 16:13:00.619409 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.619271    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e72bd4-98a9-4e48-8394-1fef1b02f73d-proxy-tls\") pod \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\" (UID: \"94e72bd4-98a9-4e48-8394-1fef1b02f73d\") "
Sep 17 16:13:00.619577 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.619555    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e72bd4-98a9-4e48-8394-1fef1b02f73d-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "94e72bd4-98a9-4e48-8394-1fef1b02f73d" (UID: "94e72bd4-98a9-4e48-8394-1fef1b02f73d"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:00.621313 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.621289    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e72bd4-98a9-4e48-8394-1fef1b02f73d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94e72bd4-98a9-4e48-8394-1fef1b02f73d" (UID: "94e72bd4-98a9-4e48-8394-1fef1b02f73d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:13:00.720180 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.720079    2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94e72bd4-98a9-4e48-8394-1fef1b02f73d-openshift-service-ca-bundle\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:13:00.720180 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:00.720123    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94e72bd4-98a9-4e48-8394-1fef1b02f73d-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:13:01.506833 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:01.506802    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"
Sep 17 16:13:01.531674 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:01.531646    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"]
Sep 17 16:13:01.541997 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:01.541973    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-raw-hpa-aa09f-686678f68d-ndpqd"]
Sep 17 16:13:02.370549 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:02.370513    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" path="/var/lib/kubelet/pods/94e72bd4-98a9-4e48-8394-1fef1b02f73d/volumes"
Sep 17 16:13:04.500120 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:04.500092    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:13:04.500744 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:04.500711    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:13:04.501095 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:04.501069    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:13:14.501011 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:14.500963    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:13:14.501456 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:14.501433    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:13:24.500962 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:24.500922    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:13:24.501458 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:24.501377    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:13:34.500723 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:34.500683    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:13:34.501172 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:34.501145    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:13:35.583174 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:35.583132    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:35.658270 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:35.658233    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:44.500988 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:44.500940    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:13:44.501431 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:44.501408    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:13:53.613042 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:53.613003    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:53.711572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:53.711531    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:54.501099 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:54.501055    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:13:54.501577 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:54.501554    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:13:58.762580 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:58.762542    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:13:59.276044 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:13:59.276006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:03.808466 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:03.808427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:04.500652 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:04.500600    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:14:04.501031 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:04.501005    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:14:08.997257 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:08.997221    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:09.151464 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:09.151429    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:11.366965 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:11.366929    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:14:11.367375 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:11.367067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:14:25.612077 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.612043    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r_c6ca8f35-3a5c-4d6a-b87c-841d7ec20424/kserve-container/0.log"
Sep 17 16:14:25.785622 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.785591    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"]
Sep 17 16:14:25.785924 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.785878    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" containerID="cri-o://caf183681ff2c1674e9adf217f2cb8217177706e0fbdaed05bbe68befa95e75f" gracePeriod=30
Sep 17 16:14:25.786094 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.785923    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" containerID="cri-o://3b39152cc6414d37ad7a3058190fee372983cbddd8771d59d0c6ef1f361e01ab" gracePeriod=30
Sep 17 16:14:25.786094 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.785923    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy" containerID="cri-o://27c1d2609694247067f4ac60179ac7b41c22c69217fdbdba03e13fe9a9136a6e" gracePeriod=30
Sep 17 16:14:25.817087 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.817048    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:25.822489 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.822461    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"]
Sep 17 16:14:25.822707 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.822677    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerName="kserve-container" containerID="cri-o://abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161" gracePeriod=30
Sep 17 16:14:25.822860 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.822720    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerName="kube-rbac-proxy" containerID="cri-o://3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec" gracePeriod=30
Sep 17 16:14:25.909610 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.909539    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:25.980053 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:25.980010    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:26.070755 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.070726    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:14:26.077541 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.077520    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:14:26.155498 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.155461    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-rbac-proxy-sar-config\") pod \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") "
Sep 17 16:14:26.155695 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.155518    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b774l\" (UniqueName: \"kubernetes.io/projected/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-api-access-b774l\") pod \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") "
Sep 17 16:14:26.155695 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.155542    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls\") pod \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\" (UID: \"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424\") "
Sep 17 16:14:26.155939 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.155911    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" (UID: "c6ca8f35-3a5c-4d6a-b87c-841d7ec20424"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:14:26.157705 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.157683    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" (UID: "c6ca8f35-3a5c-4d6a-b87c-841d7ec20424"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:14:26.157758 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.157721    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-api-access-b774l" (OuterVolumeSpecName: "kube-api-access-b774l") pod "c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" (UID: "c6ca8f35-3a5c-4d6a-b87c-841d7ec20424"). InnerVolumeSpecName "kube-api-access-b774l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:14:26.256986 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.256901    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:14:26.256986 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.256930    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b774l\" (UniqueName: \"kubernetes.io/projected/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-kube-api-access-b774l\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:14:26.256986 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.256939    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:14:26.814404 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.814369    2584 generic.go:356] "Generic (PLEG): container finished" podID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerID="3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec" exitCode=2
Sep 17 16:14:26.814404 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.814397    2584 generic.go:356] "Generic (PLEG): container finished" podID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerID="abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161" exitCode=2
Sep 17 16:14:26.814874 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.814426    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"
Sep 17 16:14:26.814874 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.814448    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" event={"ID":"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424","Type":"ContainerDied","Data":"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec"}
Sep 17 16:14:26.814874 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.814505    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" event={"ID":"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424","Type":"ContainerDied","Data":"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161"}
Sep 17 16:14:26.814874 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.814526    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r" event={"ID":"c6ca8f35-3a5c-4d6a-b87c-841d7ec20424","Type":"ContainerDied","Data":"c44f3e14bf4938e78f01ca92d8ac871efca0a63ccd474a413336735665a2df1e"}
Sep 17 16:14:26.814874 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.814542    2584 scope.go:117] "RemoveContainer" containerID="3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec"
Sep 17 16:14:26.817741 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.817719    2584 generic.go:356] "Generic (PLEG): container finished" podID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerID="27c1d2609694247067f4ac60179ac7b41c22c69217fdbdba03e13fe9a9136a6e" exitCode=2
Sep 17 16:14:26.817891 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.817797    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerDied","Data":"27c1d2609694247067f4ac60179ac7b41c22c69217fdbdba03e13fe9a9136a6e"}
Sep 17 16:14:26.822920 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.822897    2584 scope.go:117] "RemoveContainer" containerID="abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161"
Sep 17 16:14:26.830648 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.830630    2584 scope.go:117] "RemoveContainer" containerID="3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec"
Sep 17 16:14:26.831002 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:14:26.830980    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec\": container with ID starting with 3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec not found: ID does not exist" containerID="3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec"
Sep 17 16:14:26.831080 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.831008    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec"} err="failed to get container status \"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec\": rpc error: code = NotFound desc = could not find container \"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec\": container with ID starting with 3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec not found: ID does not exist"
Sep 17 16:14:26.831080 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.831025    2584 scope.go:117] "RemoveContainer" containerID="abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161"
Sep 17 16:14:26.831299 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:14:26.831282    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161\": container with ID starting with abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161 not found: ID does not exist" containerID="abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161"
Sep 17 16:14:26.831361 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.831308    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161"} err="failed to get container status \"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161\": rpc error: code = NotFound desc = could not find container \"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161\": container with ID starting with abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161 not found: ID does not exist"
Sep 17 16:14:26.831361 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.831331    2584 scope.go:117] "RemoveContainer" containerID="3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec"
Sep 17 16:14:26.831609 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.831591    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec"} err="failed to get container status \"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec\": rpc error: code = NotFound desc = could not find container \"3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec\": container with ID starting with 3bff008a0cedfc02fb4466cecab3aabc7348d82946cb21559828afdaac9359ec not found: ID does not exist"
Sep 17 16:14:26.831609 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.831610    2584 scope.go:117] "RemoveContainer" containerID="abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161"
Sep 17 16:14:26.831860 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.831839    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161"} err="failed to get container status \"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161\": rpc error: code = NotFound desc = could not find container \"abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161\": container with ID starting with abc09ac5c4689a314769a111a442807cf1c0ecff5a588a5b98f664f83deb9161 not found: ID does not exist"
Sep 17 16:14:26.835276 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.835252    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"]
Sep 17 16:14:26.839737 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:26.839713    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2b3c5-predictor-78d64b48fc-mxq2r"]
Sep 17 16:14:28.375249 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:28.375214    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" path="/var/lib/kubelet/pods/c6ca8f35-3a5c-4d6a-b87c-841d7ec20424/volumes"
Sep 17 16:14:29.496259 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:29.496218    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 16:14:30.835612 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:30.835577    2584 generic.go:356] "Generic (PLEG): container finished" podID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerID="caf183681ff2c1674e9adf217f2cb8217177706e0fbdaed05bbe68befa95e75f" exitCode=0
Sep 17 16:14:30.836013 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:30.835650    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerDied","Data":"caf183681ff2c1674e9adf217f2cb8217177706e0fbdaed05bbe68befa95e75f"}
Sep 17 16:14:31.366582 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:31.366531    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:14:31.366874 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:31.366849    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:14:34.497250 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:34.497201    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 16:14:39.497221 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:39.497181    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 16:14:39.497628 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:39.497302    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:14:41.366668 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:41.366622    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:14:41.367076 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:41.366981    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:14:44.497030 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:44.496982    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 16:14:49.496450 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:49.496401    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 16:14:51.366598 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:51.366542    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 17 16:14:51.367085 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:51.366691    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:14:51.367085 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:51.366910    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:14:51.367085 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:51.367038    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:14:54.496948 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:54.496906    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 17 16:14:55.928131 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:55.928102    2584 generic.go:356] "Generic (PLEG): container finished" podID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerID="3b39152cc6414d37ad7a3058190fee372983cbddd8771d59d0c6ef1f361e01ab" exitCode=137
Sep 17 16:14:55.928475 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:55.928172    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerDied","Data":"3b39152cc6414d37ad7a3058190fee372983cbddd8771d59d0c6ef1f361e01ab"}
Sep 17 16:14:55.967107 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:55.967084    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:14:56.078739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.078715    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kserve-provision-location\") pod \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") "
Sep 17 16:14:56.078913 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.078752    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-rbac-proxy-sar-config\") pod \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") "
Sep 17 16:14:56.078913 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.078813    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7d5zp\" (UniqueName: \"kubernetes.io/projected/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-api-access-7d5zp\") pod \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") "
Sep 17 16:14:56.078913 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.078835    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-cabundle-cert\") pod \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") "
Sep 17 16:14:56.078913 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.078877    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls\") pod \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\" (UID: \"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829\") "
Sep 17 16:14:56.079113 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.079083    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" (UID: "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:14:56.079231 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.079209    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" (UID: "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:14:56.079293 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.079243    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" (UID: "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:14:56.081087 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.081062    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-api-access-7d5zp" (OuterVolumeSpecName: "kube-api-access-7d5zp") pod "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" (UID: "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829"). InnerVolumeSpecName "kube-api-access-7d5zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:14:56.081183 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.081072    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" (UID: "52f5f061-e8f6-45cd-ba4d-26a0f0ddc829"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:14:56.179884 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.179846    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d5zp\" (UniqueName: \"kubernetes.io/projected/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-api-access-7d5zp\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:14:56.179884 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.179876    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:14:56.179884 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.179886    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:14:56.179884 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.179893    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:14:56.180141 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.179902    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:14:56.934276 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.934242    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7" event={"ID":"52f5f061-e8f6-45cd-ba4d-26a0f0ddc829","Type":"ContainerDied","Data":"6cd12203c84e0c1c765f222d76209e7ce374cbcca1e095023180a1d91ed57709"}
Sep 17 16:14:56.934707 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.934312    2584 scope.go:117] "RemoveContainer" containerID="3b39152cc6414d37ad7a3058190fee372983cbddd8771d59d0c6ef1f361e01ab"
Sep 17 16:14:56.934707 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.934254    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"
Sep 17 16:14:56.943925 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.943906    2584 scope.go:117] "RemoveContainer" containerID="27c1d2609694247067f4ac60179ac7b41c22c69217fdbdba03e13fe9a9136a6e"
Sep 17 16:14:56.950932 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.950914    2584 scope.go:117] "RemoveContainer" containerID="caf183681ff2c1674e9adf217f2cb8217177706e0fbdaed05bbe68befa95e75f"
Sep 17 16:14:56.953870 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.953841    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"]
Sep 17 16:14:56.958133 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.958113    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2b3c5-predictor-c54454df-mmnm7"]
Sep 17 16:14:56.958682 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:56.958666    2584 scope.go:117] "RemoveContainer" containerID="3b2875f4e800c09e843ab7531527b30cb3888525eb7490fe621ab4b659ab3bc8"
Sep 17 16:14:58.370426 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:14:58.370388    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" path="/var/lib/kubelet/pods/52f5f061-e8f6-45cd-ba4d-26a0f0ddc829/volumes"
Sep 17 16:15:33.749561 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:33.749516    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:33.806623 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:33.806583    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:39.206525 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:39.206481    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:44.289470 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.289433    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:44.374301 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374266    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"]
Sep 17 16:15:44.374514 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374501    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerName="kube-rbac-proxy"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374517    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerName="kube-rbac-proxy"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374526    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374531    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374540    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374545    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374553    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="storage-initializer"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374558    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="storage-initializer"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374569    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerName="kserve-container"
Sep 17 16:15:44.374572 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374574    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerName="kserve-container"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374582    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374588    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374597    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374602    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374640    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerName="kserve-container"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374647    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="agent"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374652    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6ca8f35-3a5c-4d6a-b87c-841d7ec20424" containerName="kube-rbac-proxy"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374657    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kube-rbac-proxy"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374663    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e72bd4-98a9-4e48-8394-1fef1b02f73d" containerName="model-chainer-raw-hpa-aa09f"
Sep 17 16:15:44.374905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.374669    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="52f5f061-e8f6-45cd-ba4d-26a0f0ddc829" containerName="kserve-container"
Sep 17 16:15:44.377567 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.377551    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.380382 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.380358    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-01e56-predictor-serving-cert\""
Sep 17 16:15:44.381385 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.381362    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:15:44.381481 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.381393    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:15:44.381481 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.381408    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-01e56-kube-rbac-proxy-sar-config\""
Sep 17 16:15:44.381481 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.381407    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:15:44.381481 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.381437    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:15:44.381481 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.381394    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z8dwv\""
Sep 17 16:15:44.390271 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.390243    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"]
Sep 17 16:15:44.425932 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.425900    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51637f4b-673b-4a04-b633-bb40311379de-proxy-tls\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.425932 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.425937    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jb9x\" (UniqueName: \"kubernetes.io/projected/51637f4b-673b-4a04-b633-bb40311379de-kube-api-access-2jb9x\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.426237 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.425963    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.426237 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.425995    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51637f4b-673b-4a04-b633-bb40311379de-kserve-provision-location\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.426237 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.426028    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-cabundle-cert\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.430120 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.430082    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:15:44.526497 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.526470    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51637f4b-673b-4a04-b633-bb40311379de-proxy-tls\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.526497 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.526503    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jb9x\" (UniqueName: \"kubernetes.io/projected/51637f4b-673b-4a04-b633-bb40311379de-kube-api-access-2jb9x\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.526739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.526534    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.526739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.526565    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51637f4b-673b-4a04-b633-bb40311379de-kserve-provision-location\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.526739 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.526607    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-cabundle-cert\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.527045 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.527021    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51637f4b-673b-4a04-b633-bb40311379de-kserve-provision-location\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.527361 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.527340    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-cabundle-cert\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.527433 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.527343    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.528884 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.528863    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51637f4b-673b-4a04-b633-bb40311379de-proxy-tls\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.540623 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.540570    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jb9x\" (UniqueName: \"kubernetes.io/projected/51637f4b-673b-4a04-b633-bb40311379de-kube-api-access-2jb9x\") pod \"raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.688109 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.688068    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:44.962527 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:44.962388    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"]
Sep 17 16:15:44.964862 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:15:44.964827    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51637f4b_673b_4a04_b633_bb40311379de.slice/crio-257ca87aa68c27fe983d4000b7055589b1c19e064b9175ba8b46111bae070087 WatchSource:0}: Error finding container 257ca87aa68c27fe983d4000b7055589b1c19e064b9175ba8b46111bae070087: Status 404 returned error can't find the container with id 257ca87aa68c27fe983d4000b7055589b1c19e064b9175ba8b46111bae070087
Sep 17 16:15:45.099714 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:45.099681    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" event={"ID":"51637f4b-673b-4a04-b633-bb40311379de","Type":"ContainerStarted","Data":"d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea"}
Sep 17 16:15:45.099929 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:45.099746    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" event={"ID":"51637f4b-673b-4a04-b633-bb40311379de","Type":"ContainerStarted","Data":"257ca87aa68c27fe983d4000b7055589b1c19e064b9175ba8b46111bae070087"}
Sep 17 16:15:49.117026 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:49.116941    2584 generic.go:356] "Generic (PLEG): container finished" podID="51637f4b-673b-4a04-b633-bb40311379de" containerID="d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea" exitCode=0
Sep 17 16:15:49.117026 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:49.117005    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" event={"ID":"51637f4b-673b-4a04-b633-bb40311379de","Type":"ContainerDied","Data":"d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea"}
Sep 17 16:15:50.122619 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:50.122586    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" event={"ID":"51637f4b-673b-4a04-b633-bb40311379de","Type":"ContainerStarted","Data":"0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38"}
Sep 17 16:15:50.123012 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:50.122644    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" event={"ID":"51637f4b-673b-4a04-b633-bb40311379de","Type":"ContainerStarted","Data":"e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c"}
Sep 17 16:15:50.123012 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:50.122921    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:50.123126 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:50.123083    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:50.124338 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:50.124311    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:15:50.144904 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:50.144850    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podStartSLOduration=2.345725261 podStartE2EDuration="6.14483572s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.799110459s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:50.144146797 +0000 UTC m=+1308.370816127" watchObservedRunningTime="2026-09-17 16:15:50.14483572 +0000 UTC m=+1308.371505007"
Sep 17 16:15:51.126000 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:51.125958    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:15:56.131939 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:56.131913    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:15:56.132415 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:15:56.132390    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:16:06.132881 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:16:06.132842    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:16:16.133207 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:16:16.133166    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:16:26.132865 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:16:26.132763    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:16:33.725481 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:16:33.725443    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:36.132493 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:16:36.132451    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:16:36.651944 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:16:36.651906    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:16:46.132392 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:16:46.132348    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:16:56.132742 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:16:56.132707    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 16:17:04.370600 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:04.370569    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:17:06.052292 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:06.052248    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:11.156029 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.155990    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:11.195926 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.195895    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"]
Sep 17 16:17:11.199175 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.199154    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.201732 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.201714    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 16:17:11.202002 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.201986    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 17 16:17:11.212262 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.212233    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"]
Sep 17 16:17:11.239722 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.239685    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:11.283342 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.283309    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"]
Sep 17 16:17:11.286599 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.286579    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.289066 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.289046    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 16:17:11.301886 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.301857    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"]
Sep 17 16:17:11.341188 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.341154    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.341188 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.341187    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a29a1f-c22a-4283-9a85-84306faeb84d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.341422 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.341216    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a29a1f-c22a-4283-9a85-84306faeb84d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.341422 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.341292    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.341422 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.341385    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwh6f\" (UniqueName: \"kubernetes.io/projected/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-api-access-kwh6f\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.398360 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.398328    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:11.442568 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442475    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a06fef6-2a8c-48f4-9153-61416e4d2865-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.442568 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442524    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.442568 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442549    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh64g\" (UniqueName: \"kubernetes.io/projected/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-api-access-gh64g\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.442894 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442618    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwh6f\" (UniqueName: \"kubernetes.io/projected/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-api-access-kwh6f\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.442894 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442653    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.442894 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442699    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.442894 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442736    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.442894 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a29a1f-c22a-4283-9a85-84306faeb84d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.442894 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442855    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a06fef6-2a8c-48f4-9153-61416e4d2865-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.442894 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.442889    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a29a1f-c22a-4283-9a85-84306faeb84d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.443264 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.443190    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.443264 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.443257    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a29a1f-c22a-4283-9a85-84306faeb84d-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.443418 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.443398    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.445319 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.445298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a29a1f-c22a-4283-9a85-84306faeb84d-proxy-tls\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.451814 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.451768    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwh6f\" (UniqueName: \"kubernetes.io/projected/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-api-access-kwh6f\") pod \"isvc-canary-create-v2-predictor-cdb97449-qw8mr\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.501845 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.501804    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:11.508713 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.508691    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:11.544064 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.544032    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh64g\" (UniqueName: \"kubernetes.io/projected/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-api-access-gh64g\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.544228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.544074    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.544228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.544100    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.544228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.544127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a06fef6-2a8c-48f4-9153-61416e4d2865-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.544228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.544156    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a06fef6-2a8c-48f4-9153-61416e4d2865-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.544616 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.544592    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a06fef6-2a8c-48f4-9153-61416e4d2865-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.544922 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.544895    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.545019 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.544918    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-cabundle-cert\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.546562 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.546546    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a06fef6-2a8c-48f4-9153-61416e4d2865-proxy-tls\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.553922 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.553886    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh64g\" (UniqueName: \"kubernetes.io/projected/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-api-access-gh64g\") pod \"isvc-canary-create-predictor-66d85d8bfb-pkk2g\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.596315 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.596280    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:11.788532 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.788458    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"]
Sep 17 16:17:11.792008 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:17:11.791981    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a29a1f_c22a_4283_9a85_84306faeb84d.slice/crio-c419206f104f23359174db17864a9b9f807395b3cb97803115e5e0976cd1da03 WatchSource:0}: Error finding container c419206f104f23359174db17864a9b9f807395b3cb97803115e5e0976cd1da03: Status 404 returned error can't find the container with id c419206f104f23359174db17864a9b9f807395b3cb97803115e5e0976cd1da03
Sep 17 16:17:11.867434 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:11.867403    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"]
Sep 17 16:17:11.870217 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:17:11.870191    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a06fef6_2a8c_48f4_9153_61416e4d2865.slice/crio-2299653874adecbeeeea3329cd73bbe75a93a00756e46317f50f3862eff2d962 WatchSource:0}: Error finding container 2299653874adecbeeeea3329cd73bbe75a93a00756e46317f50f3862eff2d962: Status 404 returned error can't find the container with id 2299653874adecbeeeea3329cd73bbe75a93a00756e46317f50f3862eff2d962
Sep 17 16:17:12.408245 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:12.408145    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" event={"ID":"09a29a1f-c22a-4283-9a85-84306faeb84d","Type":"ContainerStarted","Data":"0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f"}
Sep 17 16:17:12.408245 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:12.408228    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" event={"ID":"09a29a1f-c22a-4283-9a85-84306faeb84d","Type":"ContainerStarted","Data":"c419206f104f23359174db17864a9b9f807395b3cb97803115e5e0976cd1da03"}
Sep 17 16:17:12.409756 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:12.409728    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" event={"ID":"0a06fef6-2a8c-48f4-9153-61416e4d2865","Type":"ContainerStarted","Data":"f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4"}
Sep 17 16:17:12.409890 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:12.409810    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" event={"ID":"0a06fef6-2a8c-48f4-9153-61416e4d2865","Type":"ContainerStarted","Data":"2299653874adecbeeeea3329cd73bbe75a93a00756e46317f50f3862eff2d962"}
Sep 17 16:17:14.354071 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:14.354041    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"]
Sep 17 16:17:14.354457 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:14.354277    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container" containerID="cri-o://e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c" gracePeriod=30
Sep 17 16:17:14.354457 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:14.354321    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kube-rbac-proxy" containerID="cri-o://0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38" gracePeriod=30
Sep 17 16:17:14.368197 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:14.368171    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:17:15.423331 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:15.423293    2584 generic.go:356] "Generic (PLEG): container finished" podID="51637f4b-673b-4a04-b633-bb40311379de" containerID="0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38" exitCode=2
Sep 17 16:17:15.423835 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:15.423366    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" event={"ID":"51637f4b-673b-4a04-b633-bb40311379de","Type":"ContainerDied","Data":"0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38"}
Sep 17 16:17:16.126649 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:16.126601    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 17 16:17:16.428153 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:16.428120    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerID="f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4" exitCode=0
Sep 17 16:17:16.428598 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:16.428194    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" event={"ID":"0a06fef6-2a8c-48f4-9153-61416e4d2865","Type":"ContainerDied","Data":"f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4"}
Sep 17 16:17:16.429772 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:16.429752    2584 generic.go:356] "Generic (PLEG): container finished" podID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerID="0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f" exitCode=0
Sep 17 16:17:16.429881 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:16.429804    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" event={"ID":"09a29a1f-c22a-4283-9a85-84306faeb84d","Type":"ContainerDied","Data":"0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f"}
Sep 17 16:17:17.435908 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.435875    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" event={"ID":"09a29a1f-c22a-4283-9a85-84306faeb84d","Type":"ContainerStarted","Data":"83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b"}
Sep 17 16:17:17.436379 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.435948    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" event={"ID":"09a29a1f-c22a-4283-9a85-84306faeb84d","Type":"ContainerStarted","Data":"67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6"}
Sep 17 16:17:17.436379 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.436169    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:17.437944 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.437924    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" event={"ID":"0a06fef6-2a8c-48f4-9153-61416e4d2865","Type":"ContainerStarted","Data":"e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e"}
Sep 17 16:17:17.438046 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.437963    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" event={"ID":"0a06fef6-2a8c-48f4-9153-61416e4d2865","Type":"ContainerStarted","Data":"240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f"}
Sep 17 16:17:17.438152 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.438133    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:17.438196 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.438185    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:17.439577 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.439551    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:17:17.455872 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.455830    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podStartSLOduration=2.703991896 podStartE2EDuration="6.455815135s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.751823239s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:17.454384828 +0000 UTC m=+1395.681054135" watchObservedRunningTime="2026-09-17 16:17:17.455815135 +0000 UTC m=+1395.682484483"
Sep 17 16:17:17.473730 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:17.473675    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podStartSLOduration=2.5164295279999997 podStartE2EDuration="6.47365884s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.957229312s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:17.47253311 +0000 UTC m=+1395.699202420" watchObservedRunningTime="2026-09-17 16:17:17.47365884 +0000 UTC m=+1395.700328183"
Sep 17 16:17:18.442723 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:18.442682    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:17:18.443153 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:18.442880    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:18.444242 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:18.444164    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:17:19.446492 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:19.446453    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:17:19.998187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:19.998160    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:17:20.108741 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.108641    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-kube-rbac-proxy-sar-config\") pod \"51637f4b-673b-4a04-b633-bb40311379de\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") "
Sep 17 16:17:20.108741 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.108709    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jb9x\" (UniqueName: \"kubernetes.io/projected/51637f4b-673b-4a04-b633-bb40311379de-kube-api-access-2jb9x\") pod \"51637f4b-673b-4a04-b633-bb40311379de\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") "
Sep 17 16:17:20.109001 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.108809    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51637f4b-673b-4a04-b633-bb40311379de-proxy-tls\") pod \"51637f4b-673b-4a04-b633-bb40311379de\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") "
Sep 17 16:17:20.109001 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.108839    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51637f4b-673b-4a04-b633-bb40311379de-kserve-provision-location\") pod \"51637f4b-673b-4a04-b633-bb40311379de\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") "
Sep 17 16:17:20.109001 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.108873    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-cabundle-cert\") pod \"51637f4b-673b-4a04-b633-bb40311379de\" (UID: \"51637f4b-673b-4a04-b633-bb40311379de\") "
Sep 17 16:17:20.109139 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.109064    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "51637f4b-673b-4a04-b633-bb40311379de" (UID: "51637f4b-673b-4a04-b633-bb40311379de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:20.109271 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.109244    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51637f4b-673b-4a04-b633-bb40311379de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51637f4b-673b-4a04-b633-bb40311379de" (UID: "51637f4b-673b-4a04-b633-bb40311379de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:17:20.109335 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.109275    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51637f4b-673b-4a04-b633-bb40311379de" (UID: "51637f4b-673b-4a04-b633-bb40311379de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:20.110931 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.110911    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51637f4b-673b-4a04-b633-bb40311379de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51637f4b-673b-4a04-b633-bb40311379de" (UID: "51637f4b-673b-4a04-b633-bb40311379de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:20.110931 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.110926    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51637f4b-673b-4a04-b633-bb40311379de-kube-api-access-2jb9x" (OuterVolumeSpecName: "kube-api-access-2jb9x") pod "51637f4b-673b-4a04-b633-bb40311379de" (UID: "51637f4b-673b-4a04-b633-bb40311379de"). InnerVolumeSpecName "kube-api-access-2jb9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:20.209770 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.209733    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:17:20.209770 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.209766    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51637f4b-673b-4a04-b633-bb40311379de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:17:20.209770 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.209776    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jb9x\" (UniqueName: \"kubernetes.io/projected/51637f4b-673b-4a04-b633-bb40311379de-kube-api-access-2jb9x\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:17:20.210033 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.209802    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51637f4b-673b-4a04-b633-bb40311379de-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:17:20.210033 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.209811    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51637f4b-673b-4a04-b633-bb40311379de-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:17:20.452170 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.452133    2584 generic.go:356] "Generic (PLEG): container finished" podID="51637f4b-673b-4a04-b633-bb40311379de" containerID="e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c" exitCode=0
Sep 17 16:17:20.452587 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.452196    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"
Sep 17 16:17:20.452587 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.452216    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" event={"ID":"51637f4b-673b-4a04-b633-bb40311379de","Type":"ContainerDied","Data":"e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c"}
Sep 17 16:17:20.452587 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.452276    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v" event={"ID":"51637f4b-673b-4a04-b633-bb40311379de","Type":"ContainerDied","Data":"257ca87aa68c27fe983d4000b7055589b1c19e064b9175ba8b46111bae070087"}
Sep 17 16:17:20.452587 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.452301    2584 scope.go:117] "RemoveContainer" containerID="0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38"
Sep 17 16:17:20.459997 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.459985    2584 scope.go:117] "RemoveContainer" containerID="e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c"
Sep 17 16:17:20.467017 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.466999    2584 scope.go:117] "RemoveContainer" containerID="d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea"
Sep 17 16:17:20.472397 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.472375    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"]
Sep 17 16:17:20.474471 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.474455    2584 scope.go:117] "RemoveContainer" containerID="0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38"
Sep 17 16:17:20.474755 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:17:20.474733    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38\": container with ID starting with 0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38 not found: ID does not exist" containerID="0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38"
Sep 17 16:17:20.474825 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.474763    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38"} err="failed to get container status \"0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38\": rpc error: code = NotFound desc = could not find container \"0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38\": container with ID starting with 0a01ff3e2153ec9cfddd839d83f279dc06cc2d4aa8011201cbd01edb118f9b38 not found: ID does not exist"
Sep 17 16:17:20.474825 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.474805    2584 scope.go:117] "RemoveContainer" containerID="e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c"
Sep 17 16:17:20.475075 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:17:20.475056    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c\": container with ID starting with e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c not found: ID does not exist" containerID="e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c"
Sep 17 16:17:20.475111 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.475081    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c"} err="failed to get container status \"e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c\": rpc error: code = NotFound desc = could not find container \"e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c\": container with ID starting with e3d154d4925a9e2080b7b23e09c6759357b588b317cf9fc6432bd4f3dbe27b7c not found: ID does not exist"
Sep 17 16:17:20.475111 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.475097    2584 scope.go:117] "RemoveContainer" containerID="d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea"
Sep 17 16:17:20.475329 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:17:20.475312    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea\": container with ID starting with d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea not found: ID does not exist" containerID="d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea"
Sep 17 16:17:20.475368 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.475334    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea"} err="failed to get container status \"d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea\": rpc error: code = NotFound desc = could not find container \"d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea\": container with ID starting with d3cfae9281f65cdfc68cc3b414eb4626cfba76393f967172a9453e97b39bd4ea not found: ID does not exist"
Sep 17 16:17:20.477866 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:20.477847    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-01e56-predictor-86dfcf56f-79r2v"]
Sep 17 16:17:22.371937 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:22.371908    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51637f4b-673b-4a04-b633-bb40311379de" path="/var/lib/kubelet/pods/51637f4b-673b-4a04-b633-bb40311379de/volumes"
Sep 17 16:17:23.447595 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:23.447570    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:17:23.448088 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:23.448060    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:17:24.451406 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:24.451375    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:17:24.451933 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:24.451905    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:17:33.449008 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:33.448970    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:17:34.452809 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:34.452757    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:17:43.449015 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:43.448978    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:17:44.452264 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:44.452228    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:17:53.448055 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:53.448020    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:17:54.452330 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:17:54.452289    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:18:03.448140 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:03.448098    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:18:04.452392 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:04.452354    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:18:13.448201 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:13.448158    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:18:14.452506 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:14.452469    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:18:23.448224 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:23.448181    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 17 16:18:24.452866 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:24.452824    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:18:33.449478 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:33.449451    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:18:34.453437 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:34.453410    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:18:34.676954 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:34.676916    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"]
Sep 17 16:18:34.677197 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:34.677171    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container" containerID="cri-o://240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f" gracePeriod=30
Sep 17 16:18:34.677281 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:34.677206    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kube-rbac-proxy" containerID="cri-o://e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e" gracePeriod=30
Sep 17 16:18:35.744700 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:35.744667    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerID="e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e" exitCode=2
Sep 17 16:18:35.745089 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:35.744740    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" event={"ID":"0a06fef6-2a8c-48f4-9153-61416e4d2865","Type":"ContainerDied","Data":"e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e"}
Sep 17 16:18:38.443241 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:38.443203    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 17 16:18:39.415255 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.415231    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:18:39.516588 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.516484    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-cabundle-cert\") pod \"0a06fef6-2a8c-48f4-9153-61416e4d2865\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") "
Sep 17 16:18:39.516588 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.516549    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a06fef6-2a8c-48f4-9153-61416e4d2865-kserve-provision-location\") pod \"0a06fef6-2a8c-48f4-9153-61416e4d2865\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") "
Sep 17 16:18:39.517083 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.516625    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a06fef6-2a8c-48f4-9153-61416e4d2865-proxy-tls\") pod \"0a06fef6-2a8c-48f4-9153-61416e4d2865\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") "
Sep 17 16:18:39.517083 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.516649    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-rbac-proxy-sar-config\") pod \"0a06fef6-2a8c-48f4-9153-61416e4d2865\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") "
Sep 17 16:18:39.517083 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.516689    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh64g\" (UniqueName: \"kubernetes.io/projected/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-api-access-gh64g\") pod \"0a06fef6-2a8c-48f4-9153-61416e4d2865\" (UID: \"0a06fef6-2a8c-48f4-9153-61416e4d2865\") "
Sep 17 16:18:39.517083 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.516932    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0a06fef6-2a8c-48f4-9153-61416e4d2865" (UID: "0a06fef6-2a8c-48f4-9153-61416e4d2865"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:39.517083 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.516951    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a06fef6-2a8c-48f4-9153-61416e4d2865-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0a06fef6-2a8c-48f4-9153-61416e4d2865" (UID: "0a06fef6-2a8c-48f4-9153-61416e4d2865"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:18:39.517083 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.517035    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a06fef6-2a8c-48f4-9153-61416e4d2865" (UID: "0a06fef6-2a8c-48f4-9153-61416e4d2865"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:39.518790 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.518754    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a06fef6-2a8c-48f4-9153-61416e4d2865-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a06fef6-2a8c-48f4-9153-61416e4d2865" (UID: "0a06fef6-2a8c-48f4-9153-61416e4d2865"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:39.518873 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.518823    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-api-access-gh64g" (OuterVolumeSpecName: "kube-api-access-gh64g") pod "0a06fef6-2a8c-48f4-9153-61416e4d2865" (UID: "0a06fef6-2a8c-48f4-9153-61416e4d2865"). InnerVolumeSpecName "kube-api-access-gh64g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:39.617684 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.617645    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a06fef6-2a8c-48f4-9153-61416e4d2865-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:39.617684 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.617676    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:39.617684 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.617686    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh64g\" (UniqueName: \"kubernetes.io/projected/0a06fef6-2a8c-48f4-9153-61416e4d2865-kube-api-access-gh64g\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:39.617684 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.617696    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0a06fef6-2a8c-48f4-9153-61416e4d2865-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:39.617982 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.617704    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0a06fef6-2a8c-48f4-9153-61416e4d2865-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:39.762852 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.762816    2584 generic.go:356] "Generic (PLEG): container finished" podID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerID="240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f" exitCode=0
Sep 17 16:18:39.763014 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.762863    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" event={"ID":"0a06fef6-2a8c-48f4-9153-61416e4d2865","Type":"ContainerDied","Data":"240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f"}
Sep 17 16:18:39.763014 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.762874    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"
Sep 17 16:18:39.763014 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.762911    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g" event={"ID":"0a06fef6-2a8c-48f4-9153-61416e4d2865","Type":"ContainerDied","Data":"2299653874adecbeeeea3329cd73bbe75a93a00756e46317f50f3862eff2d962"}
Sep 17 16:18:39.763014 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.762947    2584 scope.go:117] "RemoveContainer" containerID="e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e"
Sep 17 16:18:39.771747 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.771725    2584 scope.go:117] "RemoveContainer" containerID="240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f"
Sep 17 16:18:39.779512 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.779495    2584 scope.go:117] "RemoveContainer" containerID="f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4"
Sep 17 16:18:39.786332 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.786315    2584 scope.go:117] "RemoveContainer" containerID="e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e"
Sep 17 16:18:39.786603 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:18:39.786586    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e\": container with ID starting with e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e not found: ID does not exist" containerID="e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e"
Sep 17 16:18:39.786650 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.786613    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e"} err="failed to get container status \"e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e\": rpc error: code = NotFound desc = could not find container \"e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e\": container with ID starting with e75e44260a47527d59c7b57bd69a4c80f3ac433c8552ee5f88095343f8024c6e not found: ID does not exist"
Sep 17 16:18:39.786650 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.786632    2584 scope.go:117] "RemoveContainer" containerID="240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f"
Sep 17 16:18:39.786892 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:18:39.786875    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f\": container with ID starting with 240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f not found: ID does not exist" containerID="240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f"
Sep 17 16:18:39.786942 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.786898    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f"} err="failed to get container status \"240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f\": rpc error: code = NotFound desc = could not find container \"240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f\": container with ID starting with 240dd9344d82aec10711edf7ba2e69958d431aa3e1c8cc717e2ade6680cef16f not found: ID does not exist"
Sep 17 16:18:39.786942 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.786913    2584 scope.go:117] "RemoveContainer" containerID="f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4"
Sep 17 16:18:39.787180 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:18:39.787163    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4\": container with ID starting with f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4 not found: ID does not exist" containerID="f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4"
Sep 17 16:18:39.787233 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.787187    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4"} err="failed to get container status \"f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4\": rpc error: code = NotFound desc = could not find container \"f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4\": container with ID starting with f5ff10f52157d6ae98e05f99c1156752a8619d0e9cf233c8ca557887d05a05b4 not found: ID does not exist"
Sep 17 16:18:39.797334 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.797312    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"]
Sep 17 16:18:39.802669 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:39.802648    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-66d85d8bfb-pkk2g"]
Sep 17 16:18:40.370803 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:40.370747    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" path="/var/lib/kubelet/pods/0a06fef6-2a8c-48f4-9153-61416e4d2865/volumes"
Sep 17 16:18:41.405135 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:41.405092    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"]
Sep 17 16:18:41.405573 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:41.405345    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" containerID="cri-o://67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6" gracePeriod=30
Sep 17 16:18:41.405573 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:41.405396    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kube-rbac-proxy" containerID="cri-o://83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b" gracePeriod=30
Sep 17 16:18:41.414463 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:41.414431    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:41.491869 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:41.491833    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:41.772506 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:41.772414    2584 generic.go:356] "Generic (PLEG): container finished" podID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerID="83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b" exitCode=2
Sep 17 16:18:41.772506 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:41.772483    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" event={"ID":"09a29a1f-c22a-4283-9a85-84306faeb84d","Type":"ContainerDied","Data":"83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b"}
Sep 17 16:18:44.447278 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:44.447233    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 16:18:44.452855 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:44.452821    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 17 16:18:46.142855 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.142832    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:18:46.164370 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.164344    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-cabundle-cert\") pod \"09a29a1f-c22a-4283-9a85-84306faeb84d\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") "
Sep 17 16:18:46.164509 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.164381    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a29a1f-c22a-4283-9a85-84306faeb84d-kserve-provision-location\") pod \"09a29a1f-c22a-4283-9a85-84306faeb84d\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") "
Sep 17 16:18:46.164509 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.164410    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kwh6f\" (UniqueName: \"kubernetes.io/projected/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-api-access-kwh6f\") pod \"09a29a1f-c22a-4283-9a85-84306faeb84d\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") "
Sep 17 16:18:46.164509 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.164430    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-rbac-proxy-sar-config\") pod \"09a29a1f-c22a-4283-9a85-84306faeb84d\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") "
Sep 17 16:18:46.164509 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.164449    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a29a1f-c22a-4283-9a85-84306faeb84d-proxy-tls\") pod \"09a29a1f-c22a-4283-9a85-84306faeb84d\" (UID: \"09a29a1f-c22a-4283-9a85-84306faeb84d\") "
Sep 17 16:18:46.164743 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.164719    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09a29a1f-c22a-4283-9a85-84306faeb84d" (UID: "09a29a1f-c22a-4283-9a85-84306faeb84d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:46.164853 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.164739    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a29a1f-c22a-4283-9a85-84306faeb84d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09a29a1f-c22a-4283-9a85-84306faeb84d" (UID: "09a29a1f-c22a-4283-9a85-84306faeb84d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:18:46.164853 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.164836    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09a29a1f-c22a-4283-9a85-84306faeb84d" (UID: "09a29a1f-c22a-4283-9a85-84306faeb84d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:46.166988 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.166950    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a29a1f-c22a-4283-9a85-84306faeb84d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09a29a1f-c22a-4283-9a85-84306faeb84d" (UID: "09a29a1f-c22a-4283-9a85-84306faeb84d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:46.167313 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.167288    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-api-access-kwh6f" (OuterVolumeSpecName: "kube-api-access-kwh6f") pod "09a29a1f-c22a-4283-9a85-84306faeb84d" (UID: "09a29a1f-c22a-4283-9a85-84306faeb84d"). InnerVolumeSpecName "kube-api-access-kwh6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:46.264934 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.264843    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:46.264934 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.264873    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09a29a1f-c22a-4283-9a85-84306faeb84d-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:46.264934 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.264882    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwh6f\" (UniqueName: \"kubernetes.io/projected/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-api-access-kwh6f\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:46.264934 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.264890    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09a29a1f-c22a-4283-9a85-84306faeb84d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:46.264934 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.264910    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09a29a1f-c22a-4283-9a85-84306faeb84d-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:18:46.443755 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.443717    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:46.531346 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.531260    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:46.537924 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.537897    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"]
Sep 17 16:18:46.538134 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538123    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="storage-initializer"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538135    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="storage-initializer"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538145    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538151    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538158    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kube-rbac-proxy"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538165    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kube-rbac-proxy"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538171    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="storage-initializer"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538176    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="storage-initializer"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538183    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container"
Sep 17 16:18:46.538187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538188    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538194    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kube-rbac-proxy"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538199    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kube-rbac-proxy"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538210    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kube-rbac-proxy"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538214    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kube-rbac-proxy"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538223    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="storage-initializer"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538228    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="storage-initializer"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538237    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538242    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538277    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kserve-container"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538284    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kserve-container"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538289    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="51637f4b-673b-4a04-b633-bb40311379de" containerName="kube-rbac-proxy"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538294    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kserve-container"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538301    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerName="kube-rbac-proxy"
Sep 17 16:18:46.538522 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.538305    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a06fef6-2a8c-48f4-9153-61416e4d2865" containerName="kube-rbac-proxy"
Sep 17 16:18:46.541328 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.541307    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.544256 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.544235    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 16:18:46.544374 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.544305    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 16:18:46.555332 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.555301    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"]
Sep 17 16:18:46.566946 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.566918    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401046a0-7919-4be6-a267-132d510698a0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.566946 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.566950    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhj5\" (UniqueName: \"kubernetes.io/projected/401046a0-7919-4be6-a267-132d510698a0-kube-api-access-tjhj5\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.567126 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.566978    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.567126 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.567055    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.567126 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.567083    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.603365 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.603331    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"]
Sep 17 16:18:46.606529 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.606512    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.609106 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.609088    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 16:18:46.618815 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.618753    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"]
Sep 17 16:18:46.667382 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667353    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.667544 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667393    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401046a0-7919-4be6-a267-132d510698a0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.667544 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667414    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhj5\" (UniqueName: \"kubernetes.io/projected/401046a0-7919-4be6-a267-132d510698a0-kube-api-access-tjhj5\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.667544 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667464    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.667544 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667515    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.667745 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667571    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.667745 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:18:46.667654    2584 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 16:18:46.667745 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:18:46.667700    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls podName:401046a0-7919-4be6-a267-132d510698a0 nodeName:}" failed. No retries permitted until 2026-09-17 16:18:47.167685129 +0000 UTC m=+1485.394354394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls") pod "isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" (UID: "401046a0-7919-4be6-a267-132d510698a0") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 17 16:18:46.667745 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667735    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.667990 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667755    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401046a0-7919-4be6-a267-132d510698a0-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.667990 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667771    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e699a51-9d57-4faa-9001-ffd6c0b6f479-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.667990 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667826    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwdst\" (UniqueName: \"kubernetes.io/projected/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-api-access-qwdst\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.667990 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.667874    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.668316 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.668294    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.668316 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.668311    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.677581 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.677557    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhj5\" (UniqueName: \"kubernetes.io/projected/401046a0-7919-4be6-a267-132d510698a0-kube-api-access-tjhj5\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:46.684352 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.684312    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:46.768798 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.768756    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.768969 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.768861    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.768969 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.768885    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e699a51-9d57-4faa-9001-ffd6c0b6f479-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.768969 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.768909    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwdst\" (UniqueName: \"kubernetes.io/projected/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-api-access-qwdst\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.768969 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.768947    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.769415 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.769391    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.769529 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.769514    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-cabundle-cert\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.769571 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.769530    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.771498 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.771476    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e699a51-9d57-4faa-9001-ffd6c0b6f479-proxy-tls\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.777366 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.777344    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwdst\" (UniqueName: \"kubernetes.io/projected/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-api-access-qwdst\") pod \"isvc-canary-promote-predictor-5fd5fb8776-msc6d\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:46.786487 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.786427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:18:46.791151 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.791129    2584 generic.go:356] "Generic (PLEG): container finished" podID="09a29a1f-c22a-4283-9a85-84306faeb84d" containerID="67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6" exitCode=0
Sep 17 16:18:46.791234 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.791213    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"
Sep 17 16:18:46.791303 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.791207    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" event={"ID":"09a29a1f-c22a-4283-9a85-84306faeb84d","Type":"ContainerDied","Data":"67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6"}
Sep 17 16:18:46.791355 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.791341    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr" event={"ID":"09a29a1f-c22a-4283-9a85-84306faeb84d","Type":"ContainerDied","Data":"c419206f104f23359174db17864a9b9f807395b3cb97803115e5e0976cd1da03"}
Sep 17 16:18:46.791396 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.791368    2584 scope.go:117] "RemoveContainer" containerID="83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b"
Sep 17 16:18:46.799003 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.798985    2584 scope.go:117] "RemoveContainer" containerID="67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6"
Sep 17 16:18:46.805846 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.805827    2584 scope.go:117] "RemoveContainer" containerID="0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f"
Sep 17 16:18:46.812598 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.812579    2584 scope.go:117] "RemoveContainer" containerID="83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b"
Sep 17 16:18:46.812921 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:18:46.812886    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b\": container with ID starting with 83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b not found: ID does not exist" containerID="83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b"
Sep 17 16:18:46.813015 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.812930    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b"} err="failed to get container status \"83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b\": rpc error: code = NotFound desc = could not find container \"83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b\": container with ID starting with 83c47ad62ad72f9384c8c6e225292dd5bafa0a3c2629c3d77c540b5021e02b7b not found: ID does not exist"
Sep 17 16:18:46.813015 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.812954    2584 scope.go:117] "RemoveContainer" containerID="67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6"
Sep 17 16:18:46.813345 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:18:46.813322    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6\": container with ID starting with 67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6 not found: ID does not exist" containerID="67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6"
Sep 17 16:18:46.813462 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.813434    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6"} err="failed to get container status \"67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6\": rpc error: code = NotFound desc = could not find container \"67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6\": container with ID starting with 67c8831880fba52d8019ab7f6de6140a49a61d6576f333103ec2cdd018a637e6 not found: ID does not exist"
Sep 17 16:18:46.813532 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.813475    2584 scope.go:117] "RemoveContainer" containerID="0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f"
Sep 17 16:18:46.813904 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:18:46.813834    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f\": container with ID starting with 0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f not found: ID does not exist" containerID="0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f"
Sep 17 16:18:46.813904 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.813866    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f"} err="failed to get container status \"0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f\": rpc error: code = NotFound desc = could not find container \"0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f\": container with ID starting with 0393c68460710f83a4c083eab511ad15683f941e524e4e1c97675af321c3077f not found: ID does not exist"
Sep 17 16:18:46.814940 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.814919    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"]
Sep 17 16:18:46.820405 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.820386    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-cdb97449-qw8mr"]
Sep 17 16:18:46.919752 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:46.919713    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:47.171298 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.171268    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:47.173922 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.173897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:47.195985 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.195963    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"]
Sep 17 16:18:47.197844 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:18:47.197816    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e699a51_9d57_4faa_9001_ffd6c0b6f479.slice/crio-4d0ad3ad30dc106c0de07cfb6af254f241a60cceb031d288497c371b66f0cc6d WatchSource:0}: Error finding container 4d0ad3ad30dc106c0de07cfb6af254f241a60cceb031d288497c371b66f0cc6d: Status 404 returned error can't find the container with id 4d0ad3ad30dc106c0de07cfb6af254f241a60cceb031d288497c371b66f0cc6d
Sep 17 16:18:47.451543 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.451454    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:47.730994 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.730920    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"]
Sep 17 16:18:47.734031 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:18:47.734004    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401046a0_7919_4be6_a267_132d510698a0.slice/crio-1afc0f1e593f2720beeb0183d7b8f28d147d0086c8b109c11ab7d775c17bbb9e WatchSource:0}: Error finding container 1afc0f1e593f2720beeb0183d7b8f28d147d0086c8b109c11ab7d775c17bbb9e: Status 404 returned error can't find the container with id 1afc0f1e593f2720beeb0183d7b8f28d147d0086c8b109c11ab7d775c17bbb9e
Sep 17 16:18:47.797769 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.797725    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" event={"ID":"401046a0-7919-4be6-a267-132d510698a0","Type":"ContainerStarted","Data":"cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d"}
Sep 17 16:18:47.797917 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.797836    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" event={"ID":"401046a0-7919-4be6-a267-132d510698a0","Type":"ContainerStarted","Data":"1afc0f1e593f2720beeb0183d7b8f28d147d0086c8b109c11ab7d775c17bbb9e"}
Sep 17 16:18:47.800362 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.800333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" event={"ID":"1e699a51-9d57-4faa-9001-ffd6c0b6f479","Type":"ContainerStarted","Data":"fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923"}
Sep 17 16:18:47.800458 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:47.800397    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" event={"ID":"1e699a51-9d57-4faa-9001-ffd6c0b6f479","Type":"ContainerStarted","Data":"4d0ad3ad30dc106c0de07cfb6af254f241a60cceb031d288497c371b66f0cc6d"}
Sep 17 16:18:48.371123 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:48.371080    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09a29a1f-c22a-4283-9a85-84306faeb84d" path="/var/lib/kubelet/pods/09a29a1f-c22a-4283-9a85-84306faeb84d/volumes"
Sep 17 16:18:51.816964 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:51.816935    2584 generic.go:356] "Generic (PLEG): container finished" podID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerID="fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923" exitCode=0
Sep 17 16:18:51.817296 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:51.817011    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" event={"ID":"1e699a51-9d57-4faa-9001-ffd6c0b6f479","Type":"ContainerDied","Data":"fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923"}
Sep 17 16:18:51.818518 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:51.818495    2584 generic.go:356] "Generic (PLEG): container finished" podID="401046a0-7919-4be6-a267-132d510698a0" containerID="cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d" exitCode=0
Sep 17 16:18:51.818614 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:51.818559    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" event={"ID":"401046a0-7919-4be6-a267-132d510698a0","Type":"ContainerDied","Data":"cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d"}
Sep 17 16:18:52.826160 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.826125    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" event={"ID":"1e699a51-9d57-4faa-9001-ffd6c0b6f479","Type":"ContainerStarted","Data":"83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425"}
Sep 17 16:18:52.826589 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.826208    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" event={"ID":"1e699a51-9d57-4faa-9001-ffd6c0b6f479","Type":"ContainerStarted","Data":"75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa"}
Sep 17 16:18:52.826589 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.826503    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:52.826693 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.826635    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:52.828124 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.828096    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:18:52.828445 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.828428    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" event={"ID":"401046a0-7919-4be6-a267-132d510698a0","Type":"ContainerStarted","Data":"cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0"}
Sep 17 16:18:52.828507 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.828466    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" event={"ID":"401046a0-7919-4be6-a267-132d510698a0","Type":"ContainerStarted","Data":"12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08"}
Sep 17 16:18:52.828696 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.828679    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:52.828768 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.828736    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:52.829649 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.829626    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:18:52.850092 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.850047    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podStartSLOduration=2.816128709 podStartE2EDuration="6.850032326s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.033903617s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:52.848988058 +0000 UTC m=+1491.075657364" watchObservedRunningTime="2026-09-17 16:18:52.850032326 +0000 UTC m=+1491.076701615"
Sep 17 16:18:52.870951 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:52.870897    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podStartSLOduration=2.867769075 podStartE2EDuration="6.870882064s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.003112989s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:52.869931646 +0000 UTC m=+1491.096600955" watchObservedRunningTime="2026-09-17 16:18:52.870882064 +0000 UTC m=+1491.097551351"
Sep 17 16:18:53.832400 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:53.832365    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:18:53.832876 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:53.832487    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:18:58.837556 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:58.837526    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:18:58.838142 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:58.837706    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:18:58.838142 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:58.838000    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:18:58.838500 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:18:58.838477    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:19:02.335293 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:02.335262    2584 scope.go:117] "RemoveContainer" containerID="fba045dc3acc6e570ff56634c7dc3ec601b38952133b5c56e4cdb51f5b1152c8"
Sep 17 16:19:08.837905 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:08.837865    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:19:08.838345 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:08.838322    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:19:18.837992 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:18.837952    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:19:18.838516 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:18.838408    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:19:28.838353 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:28.838312    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:19:28.840676 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:28.838445    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:19:38.838549 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:38.838513    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:19:38.838958 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:38.838510    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:19:47.557672 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:47.557636    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:19:48.838890 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:48.838856    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:19:48.839322 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:48.838856    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:19:58.838641 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:58.838602    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:19:58.839118 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:19:58.838602    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:20:08.838953 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:08.838923    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:20:08.839414 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:08.839018    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:20:16.651791 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:16.651758    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"]
Sep 17 16:20:16.652318 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:16.652040    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" containerID="cri-o://75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa" gracePeriod=30
Sep 17 16:20:16.652318 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:16.652058    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kube-rbac-proxy" containerID="cri-o://83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425" gracePeriod=30
Sep 17 16:20:17.135555 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:17.135514    2584 generic.go:356] "Generic (PLEG): container finished" podID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerID="83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425" exitCode=2
Sep 17 16:20:17.135736 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:17.135583    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" event={"ID":"1e699a51-9d57-4faa-9001-ffd6c0b6f479","Type":"ContainerDied","Data":"83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425"}
Sep 17 16:20:17.988752 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:17.988713    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:20:18.832761 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:18.832711    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 16:20:18.838390 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:18.838362    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 16:20:21.400123 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.400092    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:20:21.565049 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.564971    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-rbac-proxy-sar-config\") pod \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") "
Sep 17 16:20:21.565049 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.565016    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-cabundle-cert\") pod \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") "
Sep 17 16:20:21.565272 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.565060    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwdst\" (UniqueName: \"kubernetes.io/projected/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-api-access-qwdst\") pod \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") "
Sep 17 16:20:21.565272 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.565092    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kserve-provision-location\") pod \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") "
Sep 17 16:20:21.565272 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.565119    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e699a51-9d57-4faa-9001-ffd6c0b6f479-proxy-tls\") pod \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\" (UID: \"1e699a51-9d57-4faa-9001-ffd6c0b6f479\") "
Sep 17 16:20:21.565439 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.565367    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e699a51-9d57-4faa-9001-ffd6c0b6f479" (UID: "1e699a51-9d57-4faa-9001-ffd6c0b6f479"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:20:21.565439 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.565406    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1e699a51-9d57-4faa-9001-ffd6c0b6f479" (UID: "1e699a51-9d57-4faa-9001-ffd6c0b6f479"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:20:21.565439 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.565408    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1e699a51-9d57-4faa-9001-ffd6c0b6f479" (UID: "1e699a51-9d57-4faa-9001-ffd6c0b6f479"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:20:21.567315 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.567290    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e699a51-9d57-4faa-9001-ffd6c0b6f479-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e699a51-9d57-4faa-9001-ffd6c0b6f479" (UID: "1e699a51-9d57-4faa-9001-ffd6c0b6f479"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:20:21.567469 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.567453    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-api-access-qwdst" (OuterVolumeSpecName: "kube-api-access-qwdst") pod "1e699a51-9d57-4faa-9001-ffd6c0b6f479" (UID: "1e699a51-9d57-4faa-9001-ffd6c0b6f479"). InnerVolumeSpecName "kube-api-access-qwdst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:20:21.665839 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.665767    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:20:21.665839 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.665834    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1e699a51-9d57-4faa-9001-ffd6c0b6f479-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:20:21.665839 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.665850    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwdst\" (UniqueName: \"kubernetes.io/projected/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kube-api-access-qwdst\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:20:21.666148 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.665862    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1e699a51-9d57-4faa-9001-ffd6c0b6f479-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:20:21.666148 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:21.665875    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e699a51-9d57-4faa-9001-ffd6c0b6f479-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:20:22.157240 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.157205    2584 generic.go:356] "Generic (PLEG): container finished" podID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerID="75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa" exitCode=0
Sep 17 16:20:22.157499 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.157257    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"
Sep 17 16:20:22.157499 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.157284    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" event={"ID":"1e699a51-9d57-4faa-9001-ffd6c0b6f479","Type":"ContainerDied","Data":"75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa"}
Sep 17 16:20:22.157499 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.157340    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d" event={"ID":"1e699a51-9d57-4faa-9001-ffd6c0b6f479","Type":"ContainerDied","Data":"4d0ad3ad30dc106c0de07cfb6af254f241a60cceb031d288497c371b66f0cc6d"}
Sep 17 16:20:22.157499 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.157366    2584 scope.go:117] "RemoveContainer" containerID="83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425"
Sep 17 16:20:22.165503 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.165483    2584 scope.go:117] "RemoveContainer" containerID="75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa"
Sep 17 16:20:22.172292 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.172274    2584 scope.go:117] "RemoveContainer" containerID="fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923"
Sep 17 16:20:22.178166 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.178144    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"]
Sep 17 16:20:22.180131 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.180109    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-5fd5fb8776-msc6d"]
Sep 17 16:20:22.181182 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.181166    2584 scope.go:117] "RemoveContainer" containerID="83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425"
Sep 17 16:20:22.181537 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:20:22.181519    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425\": container with ID starting with 83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425 not found: ID does not exist" containerID="83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425"
Sep 17 16:20:22.181603 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.181546    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425"} err="failed to get container status \"83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425\": rpc error: code = NotFound desc = could not find container \"83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425\": container with ID starting with 83ca1b54df2f227291a251cdcaeab7044526da5fe86f90d3d13fbb47b990e425 not found: ID does not exist"
Sep 17 16:20:22.181603 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.181562    2584 scope.go:117] "RemoveContainer" containerID="75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa"
Sep 17 16:20:22.181865 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:20:22.181843    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa\": container with ID starting with 75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa not found: ID does not exist" containerID="75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa"
Sep 17 16:20:22.181946 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.181874    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa"} err="failed to get container status \"75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa\": rpc error: code = NotFound desc = could not find container \"75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa\": container with ID starting with 75b400a08c65b61cf78faed1035f9dd32fa95c0c01026b49dc7557409eee07fa not found: ID does not exist"
Sep 17 16:20:22.181946 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.181896    2584 scope.go:117] "RemoveContainer" containerID="fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923"
Sep 17 16:20:22.182213 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:20:22.182197    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923\": container with ID starting with fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923 not found: ID does not exist" containerID="fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923"
Sep 17 16:20:22.182254 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.182218    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923"} err="failed to get container status \"fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923\": rpc error: code = NotFound desc = could not find container \"fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923\": container with ID starting with fce8043b49bb11e1e64e8ceb0a32bf7b4295b8b77b600b4b84f44a1ab218d923 not found: ID does not exist"
Sep 17 16:20:22.370001 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:20:22.369971    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" path="/var/lib/kubelet/pods/1e699a51-9d57-4faa-9001-ffd6c0b6f479/volumes"
Sep 17 16:21:00.793606 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:00.793526    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:21.101391 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:21.101348    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:46.936406 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:46.936372    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"]
Sep 17 16:21:46.936960 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:46.936617    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" containerID="cri-o://12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08" gracePeriod=30
Sep 17 16:21:46.936960 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:46.936662    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kube-rbac-proxy" containerID="cri-o://cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0" gracePeriod=30
Sep 17 16:21:46.962056 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:46.962012    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:21:47.452671 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:47.452639    2584 generic.go:356] "Generic (PLEG): container finished" podID="401046a0-7919-4be6-a267-132d510698a0" containerID="cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0" exitCode=2
Sep 17 16:21:47.452884 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:47.452711    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" event={"ID":"401046a0-7919-4be6-a267-132d510698a0","Type":"ContainerDied","Data":"cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0"}
Sep 17 16:21:48.833334 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:48.833285    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 16:21:48.838680 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:48.838650    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 16:21:51.679148 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.679124    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:21:51.726008 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.725977    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls\") pod \"401046a0-7919-4be6-a267-132d510698a0\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") "
Sep 17 16:21:51.726199 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.726019    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-kube-rbac-proxy-sar-config\") pod \"401046a0-7919-4be6-a267-132d510698a0\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") "
Sep 17 16:21:51.726199 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.726053    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-cabundle-cert\") pod \"401046a0-7919-4be6-a267-132d510698a0\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") "
Sep 17 16:21:51.726199 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.726102    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401046a0-7919-4be6-a267-132d510698a0-kserve-provision-location\") pod \"401046a0-7919-4be6-a267-132d510698a0\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") "
Sep 17 16:21:51.726199 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.726132    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjhj5\" (UniqueName: \"kubernetes.io/projected/401046a0-7919-4be6-a267-132d510698a0-kube-api-access-tjhj5\") pod \"401046a0-7919-4be6-a267-132d510698a0\" (UID: \"401046a0-7919-4be6-a267-132d510698a0\") "
Sep 17 16:21:51.726459 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.726435    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401046a0-7919-4be6-a267-132d510698a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "401046a0-7919-4be6-a267-132d510698a0" (UID: "401046a0-7919-4be6-a267-132d510698a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:51.726523 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.726453    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "401046a0-7919-4be6-a267-132d510698a0" (UID: "401046a0-7919-4be6-a267-132d510698a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:21:51.726523 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.726461    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "401046a0-7919-4be6-a267-132d510698a0" (UID: "401046a0-7919-4be6-a267-132d510698a0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:21:51.728177 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.728149    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "401046a0-7919-4be6-a267-132d510698a0" (UID: "401046a0-7919-4be6-a267-132d510698a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:21:51.728177 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.728155    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401046a0-7919-4be6-a267-132d510698a0-kube-api-access-tjhj5" (OuterVolumeSpecName: "kube-api-access-tjhj5") pod "401046a0-7919-4be6-a267-132d510698a0" (UID: "401046a0-7919-4be6-a267-132d510698a0"). InnerVolumeSpecName "kube-api-access-tjhj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:21:51.826558 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.826521    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/401046a0-7919-4be6-a267-132d510698a0-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:21:51.826558 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.826555    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjhj5\" (UniqueName: \"kubernetes.io/projected/401046a0-7919-4be6-a267-132d510698a0-kube-api-access-tjhj5\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:21:51.826763 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.826568    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/401046a0-7919-4be6-a267-132d510698a0-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:21:51.826763 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.826581    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:21:51.826763 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:51.826593    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/401046a0-7919-4be6-a267-132d510698a0-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:21:52.471919 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.471883    2584 generic.go:356] "Generic (PLEG): container finished" podID="401046a0-7919-4be6-a267-132d510698a0" containerID="12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08" exitCode=0
Sep 17 16:21:52.472110 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.471939    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" event={"ID":"401046a0-7919-4be6-a267-132d510698a0","Type":"ContainerDied","Data":"12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08"}
Sep 17 16:21:52.472110 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.471948    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"
Sep 17 16:21:52.472110 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.472002    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm" event={"ID":"401046a0-7919-4be6-a267-132d510698a0","Type":"ContainerDied","Data":"1afc0f1e593f2720beeb0183d7b8f28d147d0086c8b109c11ab7d775c17bbb9e"}
Sep 17 16:21:52.472110 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.472039    2584 scope.go:117] "RemoveContainer" containerID="cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0"
Sep 17 16:21:52.481139 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.481113    2584 scope.go:117] "RemoveContainer" containerID="12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08"
Sep 17 16:21:52.488121 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.488103    2584 scope.go:117] "RemoveContainer" containerID="cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d"
Sep 17 16:21:52.494662 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.494648    2584 scope.go:117] "RemoveContainer" containerID="cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0"
Sep 17 16:21:52.494951 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:21:52.494934    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0\": container with ID starting with cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0 not found: ID does not exist" containerID="cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0"
Sep 17 16:21:52.495020 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.494958    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0"} err="failed to get container status \"cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0\": rpc error: code = NotFound desc = could not find container \"cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0\": container with ID starting with cc6c5b127439e59aa4a59246ca5c7a119c4199f0eb13be9e7960f339e29845c0 not found: ID does not exist"
Sep 17 16:21:52.495020 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.494976    2584 scope.go:117] "RemoveContainer" containerID="12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08"
Sep 17 16:21:52.495202 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:21:52.495188    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08\": container with ID starting with 12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08 not found: ID does not exist" containerID="12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08"
Sep 17 16:21:52.495242 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.495204    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08"} err="failed to get container status \"12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08\": rpc error: code = NotFound desc = could not find container \"12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08\": container with ID starting with 12194f6e9a669e6814f1cd7e40d2069304728ad9375d37f57718654d9e33cf08 not found: ID does not exist"
Sep 17 16:21:52.495242 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.495215    2584 scope.go:117] "RemoveContainer" containerID="cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d"
Sep 17 16:21:52.495464 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:21:52.495448    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d\": container with ID starting with cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d not found: ID does not exist" containerID="cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d"
Sep 17 16:21:52.495508 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.495470    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d"} err="failed to get container status \"cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d\": rpc error: code = NotFound desc = could not find container \"cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d\": container with ID starting with cd2a98f6b4357064443e149b4463faad1aafbc0e97f841b2ccaf4d1a6ad4110d not found: ID does not exist"
Sep 17 16:21:52.504273 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.504252    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"]
Sep 17 16:21:52.518488 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:52.518468    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-7c98bbb8c9-jt5pm"]
Sep 17 16:21:54.370028 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:21:54.369996    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401046a0-7919-4be6-a267-132d510698a0" path="/var/lib/kubelet/pods/401046a0-7919-4be6-a267-132d510698a0/volumes"
Sep 17 16:22:01.597429 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:01.597392    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:01.705128 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:01.705094    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:21.637171 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.637130    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:21.722202 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722170    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"]
Sep 17 16:22:21.722563 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722544    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="storage-initializer"
Sep 17 16:22:21.722630 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722566    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="storage-initializer"
Sep 17 16:22:21.722630 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722589    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kube-rbac-proxy"
Sep 17 16:22:21.722630 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722600    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kube-rbac-proxy"
Sep 17 16:22:21.722630 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722610    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="storage-initializer"
Sep 17 16:22:21.722630 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722618    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="storage-initializer"
Sep 17 16:22:21.722630 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722631    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722640    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722656    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kube-rbac-proxy"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722662    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kube-rbac-proxy"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722668    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722673    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722719    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kube-rbac-proxy"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722726    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kube-rbac-proxy"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722733    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e699a51-9d57-4faa-9001-ffd6c0b6f479" containerName="kserve-container"
Sep 17 16:22:21.722856 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.722739    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="401046a0-7919-4be6-a267-132d510698a0" containerName="kserve-container"
Sep 17 16:22:21.725827 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.725808    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.730225 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.730205    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:22:21.730320 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.730258    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:22:21.730320 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.730209    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:22:21.730920 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.730904    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:22:21.730971 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.730953    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 17 16:22:21.731558 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.731541    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 17 16:22:21.732793 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.732770    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-z8dwv\""
Sep 17 16:22:21.764935 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.764902    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"]
Sep 17 16:22:21.828998 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.828962    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e74377d-f415-43d8-ba55-85dc2af0fcb5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.828998 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.829003    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.829195 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.829027    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-api-access-8g95j\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.829195 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.829043    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.829195 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.829136    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.930200 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.930124    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e74377d-f415-43d8-ba55-85dc2af0fcb5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.930200 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.930166    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.930200 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.930196    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-api-access-8g95j\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.930446 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.930220    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.930446 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.930279    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.930681 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.930660    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.930891 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.930866    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.930957 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.930897    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.932723 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.932703    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e74377d-f415-43d8-ba55-85dc2af0fcb5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:21.952344 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:21.952321    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-api-access-8g95j\") pod \"isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:22.035617 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:22.035579    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:22.345933 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:22.345904    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"]
Sep 17 16:22:22.349752 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:22:22.349703    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e74377d_f415_43d8_ba55_85dc2af0fcb5.slice/crio-ec897e9ae2c0930ceec00aa973a2d84b92e206e6c840ea96a6dc34725b6eedda WatchSource:0}: Error finding container ec897e9ae2c0930ceec00aa973a2d84b92e206e6c840ea96a6dc34725b6eedda: Status 404 returned error can't find the container with id ec897e9ae2c0930ceec00aa973a2d84b92e206e6c840ea96a6dc34725b6eedda
Sep 17 16:22:22.576287 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:22.576252    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerStarted","Data":"7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7"}
Sep 17 16:22:22.576445 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:22.576333    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerStarted","Data":"ec897e9ae2c0930ceec00aa973a2d84b92e206e6c840ea96a6dc34725b6eedda"}
Sep 17 16:22:23.954051 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:23.954006    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:22:26.590810 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:26.590763    2584 generic.go:356] "Generic (PLEG): container finished" podID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerID="7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7" exitCode=0
Sep 17 16:22:26.591273 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:26.590837    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerDied","Data":"7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7"}
Sep 17 16:22:27.597230 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:27.597197    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerStarted","Data":"334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b"}
Sep 17 16:22:27.597613 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:27.597261    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerStarted","Data":"9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d"}
Sep 17 16:22:27.597613 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:27.597286    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerStarted","Data":"4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45"}
Sep 17 16:22:27.597613 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:27.597571    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:27.597745 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:27.597724    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:27.598926 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:27.598901    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:22:27.621998 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:27.621948    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podStartSLOduration=2.969053461 podStartE2EDuration="6.621907285s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.652853824s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:22:27.620827878 +0000 UTC m=+1705.847497189" watchObservedRunningTime="2026-09-17 16:22:27.621907285 +0000 UTC m=+1705.848576572"
Sep 17 16:22:28.601092 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:28.601067    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:28.601471 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:28.601040    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:22:28.602260 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:28.602231    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:22:29.604857 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:29.604812    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:22:29.605287 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:29.605227    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:22:34.608638 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:34.608604    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:22:34.609201 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:34.609167    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:22:34.609422 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:34.609400    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:22:44.609171 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:44.609125    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:22:44.609639 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:44.609588    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:22:54.609751 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:54.609714    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:22:54.610216 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:22:54.610194    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:23:04.609385 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:04.609338    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:23:04.609956 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:04.609883    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:23:14.609773 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:14.609729    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:23:14.610282 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:14.610153    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:23:24.609079 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:24.609027    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:23:24.609565 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:24.609522    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:23:34.609957 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:34.609914    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:23:34.615011 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:34.614980    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:23:44.609919 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:44.609887    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:23:44.610288 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:44.609974    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:23:56.839560 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:56.839481    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"]
Sep 17 16:23:56.840104 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:56.839794    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" containerID="cri-o://4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45" gracePeriod=30
Sep 17 16:23:56.840104 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:56.839809    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" containerID="cri-o://334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b" gracePeriod=30
Sep 17 16:23:56.840104 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:56.839841    2584 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy" containerID="cri-o://9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d" gracePeriod=30
Sep 17 16:23:56.899102 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:56.899069    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:56.962583 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:56.962550    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:57.910215 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:57.910184    2584 generic.go:356] "Generic (PLEG): container finished" podID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerID="9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d" exitCode=2
Sep 17 16:23:57.910606 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:57.910260    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerDied","Data":"9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d"}
Sep 17 16:23:58.720593 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:58.720559    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:58.796194 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:58.796159    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:23:59.605973 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:23:59.605933    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:24:01.925962 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:01.925925    2584 generic.go:356] "Generic (PLEG): container finished" podID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerID="4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45" exitCode=0
Sep 17 16:24:01.926342 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:01.925996    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerDied","Data":"4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45"}
Sep 17 16:24:04.605514 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:04.605468    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:24:04.609819 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:04.609766    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:24:04.610188 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:04.610166    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:24:09.605683 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:09.605642    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:24:09.606155 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:09.605817    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:24:14.466360 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:14.466329    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:14.508079 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:14.508049    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:14.521635 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:14.521611    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:14.535958 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:14.535934    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:14.605712 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:14.605670    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:24:14.609043 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:14.609016    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:24:14.609366 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:14.609345    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:24:15.385231 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:15.385199    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:15.398157 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:15.398128    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:15.409110 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:15.409088    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:15.421044 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:15.421020    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:16.265366 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:16.265335    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:16.285402 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:16.285379    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:16.298927 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:16.298898    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:16.311068 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:16.311047    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:17.154180 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:17.154145    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:17.179291 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:17.179257    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:17.192293 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:17.192265    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:17.210187 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:17.210163    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:18.094234 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:18.094205    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:18.108108 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:18.108080    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:18.118450 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:18.118420    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:18.132217 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:18.132195    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:18.976545 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:18.976517    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:18.993304 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:18.993281    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:19.018178 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:19.018156    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:19.035740 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:19.035713    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:19.605544 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:19.605501    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:24:19.889202 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:19.889111    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:19.902656 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:19.902627    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:19.916316 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:19.916292    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:19.931228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:19.931200    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:20.773580 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:20.773550    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:20.814995 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:20.814967    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:20.833359 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:20.833338    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:20.849086 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:20.849053    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:21.731775 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:21.731740    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:21.752470 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:21.752444    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:21.788287 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:21.788262    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:21.809381 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:21.809355    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:22.635414 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:22.635384    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:22.647866 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:22.647843    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:22.660295 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:22.660275    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:22.671430 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:22.671411    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:23.503601 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:23.503568    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:23.517690 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:23.517667    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:23.530731 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:23.530714    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:23.542979 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:23.542960    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:24.386931 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.386894    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:24.406310 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.406282    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:24.422237 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.422212    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:24.438902 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.438869    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:24.605516 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.605478    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 16:24:24.609049 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.609011    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 16:24:24.609182 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.609133    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:24:24.609404 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.609371    2584 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 16:24:24.609496 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:24.609484    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:24:25.278698 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:25.278667    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:25.292400 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:25.292376    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:25.305333 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:25.305311    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:25.317013 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:25.316988    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:26.194203 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:26.194166    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kserve-container/0.log"
Sep 17 16:24:26.207745 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:26.207718    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/kube-rbac-proxy/0.log"
Sep 17 16:24:26.219441 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:26.219417    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/agent/0.log"
Sep 17 16:24:26.235363 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:26.235330    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc_4e74377d-f415-43d8-ba55-85dc2af0fcb5/storage-initializer/0.log"
Sep 17 16:24:26.983940 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:26.983914    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:24:27.018032 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.017997    2584 generic.go:356] "Generic (PLEG): container finished" podID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerID="334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b" exitCode=0
Sep 17 16:24:27.018218 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.018075    2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"
Sep 17 16:24:27.018218 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.018076    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerDied","Data":"334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b"}
Sep 17 16:24:27.018218 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.018185    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc" event={"ID":"4e74377d-f415-43d8-ba55-85dc2af0fcb5","Type":"ContainerDied","Data":"ec897e9ae2c0930ceec00aa973a2d84b92e206e6c840ea96a6dc34725b6eedda"}
Sep 17 16:24:27.018397 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.018231    2584 scope.go:117] "RemoveContainer" containerID="334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b"
Sep 17 16:24:27.026032 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.026015    2584 scope.go:117] "RemoveContainer" containerID="9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d"
Sep 17 16:24:27.032745 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.032729    2584 scope.go:117] "RemoveContainer" containerID="4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45"
Sep 17 16:24:27.039326 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.039309    2584 scope.go:117] "RemoveContainer" containerID="7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7"
Sep 17 16:24:27.045757 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.045740    2584 scope.go:117] "RemoveContainer" containerID="334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b"
Sep 17 16:24:27.046074 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:24:27.046055    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b\": container with ID starting with 334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b not found: ID does not exist" containerID="334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b"
Sep 17 16:24:27.046147 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.046083    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b"} err="failed to get container status \"334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b\": rpc error: code = NotFound desc = could not find container \"334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b\": container with ID starting with 334ecf03dfddc94b2413244db7628c0cb2bde9408afaedfb142888371d22935b not found: ID does not exist"
Sep 17 16:24:27.046147 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.046100    2584 scope.go:117] "RemoveContainer" containerID="9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d"
Sep 17 16:24:27.046341 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:24:27.046325    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d\": container with ID starting with 9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d not found: ID does not exist" containerID="9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d"
Sep 17 16:24:27.046385 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.046351    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d"} err="failed to get container status \"9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d\": rpc error: code = NotFound desc = could not find container \"9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d\": container with ID starting with 9cdc6061191ee1fd408c4da92676542ffce65a6b89cf737b8337e80a2d48ac1d not found: ID does not exist"
Sep 17 16:24:27.046385 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.046370    2584 scope.go:117] "RemoveContainer" containerID="4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45"
Sep 17 16:24:27.046629 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:24:27.046613    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45\": container with ID starting with 4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45 not found: ID does not exist" containerID="4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45"
Sep 17 16:24:27.046672 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.046635    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45"} err="failed to get container status \"4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45\": rpc error: code = NotFound desc = could not find container \"4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45\": container with ID starting with 4f7a3ed890f6d7b424f1785b7818ce21ec7576cbbf49ed5ddf56c1497a158b45 not found: ID does not exist"
Sep 17 16:24:27.046672 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.046649    2584 scope.go:117] "RemoveContainer" containerID="7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7"
Sep 17 16:24:27.047010 ip-10-0-128-128 kubenswrapper[2584]: E0917 16:24:27.046992    2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7\": container with ID starting with 7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7 not found: ID does not exist" containerID="7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7"
Sep 17 16:24:27.047074 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.047013    2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7"} err="failed to get container status \"7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7\": rpc error: code = NotFound desc = could not find container \"7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7\": container with ID starting with 7486b825dfa29f82981cd64c15f48a2b0b7f6bce3821f21f70ba8f187cd83ef7 not found: ID does not exist"
Sep 17 16:24:27.129886 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.129798    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-rbac-proxy-sar-config\") pod \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") "
Sep 17 16:24:27.129886 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.129841    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e74377d-f415-43d8-ba55-85dc2af0fcb5-proxy-tls\") pod \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") "
Sep 17 16:24:27.129886 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.129863    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-cabundle-cert\") pod \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") "
Sep 17 16:24:27.129886 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.129883    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kserve-provision-location\") pod \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") "
Sep 17 16:24:27.130195 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.129909    2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-api-access-8g95j\") pod \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\" (UID: \"4e74377d-f415-43d8-ba55-85dc2af0fcb5\") "
Sep 17 16:24:27.130263 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.130235    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e74377d-f415-43d8-ba55-85dc2af0fcb5" (UID: "4e74377d-f415-43d8-ba55-85dc2af0fcb5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:24:27.130328 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.130253    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e74377d-f415-43d8-ba55-85dc2af0fcb5" (UID: "4e74377d-f415-43d8-ba55-85dc2af0fcb5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:24:27.130328 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.130239    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e74377d-f415-43d8-ba55-85dc2af0fcb5" (UID: "4e74377d-f415-43d8-ba55-85dc2af0fcb5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:24:27.132084 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.132055    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e74377d-f415-43d8-ba55-85dc2af0fcb5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e74377d-f415-43d8-ba55-85dc2af0fcb5" (UID: "4e74377d-f415-43d8-ba55-85dc2af0fcb5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:24:27.132178 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.132104    2584 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-api-access-8g95j" (OuterVolumeSpecName: "kube-api-access-8g95j") pod "4e74377d-f415-43d8-ba55-85dc2af0fcb5" (UID: "4e74377d-f415-43d8-ba55-85dc2af0fcb5"). InnerVolumeSpecName "kube-api-access-8g95j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:24:27.230512 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.230477    2584 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:24:27.230512 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.230506    2584 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e74377d-f415-43d8-ba55-85dc2af0fcb5-proxy-tls\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:24:27.230512 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.230517    2584 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e74377d-f415-43d8-ba55-85dc2af0fcb5-cabundle-cert\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:24:27.231007 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.230526    2584 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kserve-provision-location\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:24:27.231007 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.230535    2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g95j\" (UniqueName: \"kubernetes.io/projected/4e74377d-f415-43d8-ba55-85dc2af0fcb5-kube-api-access-8g95j\") on node \"ip-10-0-128-128.ec2.internal\" DevicePath \"\""
Sep 17 16:24:27.342121 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.342087    2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"]
Sep 17 16:24:27.346117 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:27.346094    2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5f4c5fb79b-6p8dc"]
Sep 17 16:24:28.370559 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:28.370527    2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" path="/var/lib/kubelet/pods/4e74377d-f415-43d8-ba55-85dc2af0fcb5/volumes"
Sep 17 16:24:30.080002 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.079972    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:30.254136 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254105    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vp8j5/must-gather-g2gfq"]
Sep 17 16:24:30.254352 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254340    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container"
Sep 17 16:24:30.254395 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254353    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container"
Sep 17 16:24:30.254395 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254369    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent"
Sep 17 16:24:30.254395 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254374    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent"
Sep 17 16:24:30.254395 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254384    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy"
Sep 17 16:24:30.254521 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254390    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy"
Sep 17 16:24:30.254521 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254419    2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="storage-initializer"
Sep 17 16:24:30.254521 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254425    2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="storage-initializer"
Sep 17 16:24:30.254521 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254462    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kserve-container"
Sep 17 16:24:30.254521 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254468    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="kube-rbac-proxy"
Sep 17 16:24:30.254521 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.254476    2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e74377d-f415-43d8-ba55-85dc2af0fcb5" containerName="agent"
Sep 17 16:24:30.257302 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.257279    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp8j5/must-gather-g2gfq"
Sep 17 16:24:30.260696 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.260669    2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vp8j5\"/\"default-dockercfg-lbwkw\""
Sep 17 16:24:30.261087 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.261068    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vp8j5\"/\"kube-root-ca.crt\""
Sep 17 16:24:30.261202 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.261134    2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vp8j5\"/\"openshift-service-ca.crt\""
Sep 17 16:24:30.263461 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.263436    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:30.272818 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.272796    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp8j5/must-gather-g2gfq"]
Sep 17 16:24:30.342576 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.342489    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:30.352266 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.352242    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c60231a2-8c71-4e44-bbc6-dcb9e4bec232-must-gather-output\") pod \"must-gather-g2gfq\" (UID: \"c60231a2-8c71-4e44-bbc6-dcb9e4bec232\") " pod="openshift-must-gather-vp8j5/must-gather-g2gfq"
Sep 17 16:24:30.352386 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.352271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntq7f\" (UniqueName: \"kubernetes.io/projected/c60231a2-8c71-4e44-bbc6-dcb9e4bec232-kube-api-access-ntq7f\") pod \"must-gather-g2gfq\" (UID: \"c60231a2-8c71-4e44-bbc6-dcb9e4bec232\") " pod="openshift-must-gather-vp8j5/must-gather-g2gfq"
Sep 17 16:24:30.453483 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.453440    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c60231a2-8c71-4e44-bbc6-dcb9e4bec232-must-gather-output\") pod \"must-gather-g2gfq\" (UID: \"c60231a2-8c71-4e44-bbc6-dcb9e4bec232\") " pod="openshift-must-gather-vp8j5/must-gather-g2gfq"
Sep 17 16:24:30.453663 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.453496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntq7f\" (UniqueName: \"kubernetes.io/projected/c60231a2-8c71-4e44-bbc6-dcb9e4bec232-kube-api-access-ntq7f\") pod \"must-gather-g2gfq\" (UID: \"c60231a2-8c71-4e44-bbc6-dcb9e4bec232\") " pod="openshift-must-gather-vp8j5/must-gather-g2gfq"
Sep 17 16:24:30.453846 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.453825    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c60231a2-8c71-4e44-bbc6-dcb9e4bec232-must-gather-output\") pod \"must-gather-g2gfq\" (UID: \"c60231a2-8c71-4e44-bbc6-dcb9e4bec232\") " pod="openshift-must-gather-vp8j5/must-gather-g2gfq"
Sep 17 16:24:30.462552 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.462519    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntq7f\" (UniqueName: \"kubernetes.io/projected/c60231a2-8c71-4e44-bbc6-dcb9e4bec232-kube-api-access-ntq7f\") pod \"must-gather-g2gfq\" (UID: \"c60231a2-8c71-4e44-bbc6-dcb9e4bec232\") " pod="openshift-must-gather-vp8j5/must-gather-g2gfq"
Sep 17 16:24:30.566319 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.566288    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp8j5/must-gather-g2gfq"
Sep 17 16:24:30.848129 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.848093    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp8j5/must-gather-g2gfq"]
Sep 17 16:24:30.851249 ip-10-0-128-128 kubenswrapper[2584]: W0917 16:24:30.851222    2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc60231a2_8c71_4e44_bbc6_dcb9e4bec232.slice/crio-ada1d4b6ff6e17a381ef3bf6d1f3e1f8ccfea86e9303f2b9f55901cb76946c56 WatchSource:0}: Error finding container ada1d4b6ff6e17a381ef3bf6d1f3e1f8ccfea86e9303f2b9f55901cb76946c56: Status 404 returned error can't find the container with id ada1d4b6ff6e17a381ef3bf6d1f3e1f8ccfea86e9303f2b9f55901cb76946c56
Sep 17 16:24:30.853204 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:30.853185    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:24:31.033083 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:31.033000    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp8j5/must-gather-g2gfq" event={"ID":"c60231a2-8c71-4e44-bbc6-dcb9e4bec232","Type":"ContainerStarted","Data":"ada1d4b6ff6e17a381ef3bf6d1f3e1f8ccfea86e9303f2b9f55901cb76946c56"}
Sep 17 16:24:32.039399 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:32.039306    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp8j5/must-gather-g2gfq" event={"ID":"c60231a2-8c71-4e44-bbc6-dcb9e4bec232","Type":"ContainerStarted","Data":"3d8fd8d8f0ecb119e8048e98b30d78f11d791a4feb0d94adf02a5b6a5a02128b"}
Sep 17 16:24:32.039399 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:32.039355    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp8j5/must-gather-g2gfq" event={"ID":"c60231a2-8c71-4e44-bbc6-dcb9e4bec232","Type":"ContainerStarted","Data":"fb8b0fbf014d34e0fc9bac5448075e1edfa6c8cdacf9642e33735774ca5feb42"}
Sep 17 16:24:33.151101 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:33.151066    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w8xbr_a50581ee-e188-4fad-8c5b-96896f8e15b4/global-pull-secret-syncer/0.log"
Sep 17 16:24:33.198134 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:33.198104    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log"
Sep 17 16:24:33.294293 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:33.294264    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-128.ec2.internal_42f9cee7deb9c35203a248ac01b72173/haproxy/0.log"
Sep 17 16:24:36.059363 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:36.059332    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:36.129068 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:36.129021    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c8x87_42747b31-db23-4a26-85cd-ccd997d13d0c/konnectivity-agent/0.log" limit=1024
Sep 17 16:24:37.452559 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:37.452474    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rr98r_f0971868-78f4-4808-a898-a9668294ed59/node-exporter/0.log"
Sep 17 16:24:37.483228 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:37.483166    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rr98r_f0971868-78f4-4808-a898-a9668294ed59/kube-rbac-proxy/0.log"
Sep 17 16:24:37.521252 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:37.521149    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rr98r_f0971868-78f4-4808-a898-a9668294ed59/init-textfile/0.log"
Sep 17 16:24:41.129466 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.129379    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vp8j5/must-gather-g2gfq" podStartSLOduration=10.368563953 podStartE2EDuration="11.129348603s" totalImagesPullingTime="760.78465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:24:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:24:32.057433509 +0000 UTC m=+1830.284102796" watchObservedRunningTime="2026-09-17 16:24:41.129348603 +0000 UTC m=+1839.356017892"
Sep 17 16:24:41.130082 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.129552    2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"]
Sep 17 16:24:41.135248 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.135224    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.154240 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.154160    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"]
Sep 17 16:24:41.242886 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.242844    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lgv\" (UniqueName: \"kubernetes.io/projected/12a055f8-f29e-4c08-8f41-e11f693065ab-kube-api-access-v4lgv\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.243080 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.242899    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-sys\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.243080 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.242958    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-podres\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.243080 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.243003    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-proc\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.243080 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.243019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-lib-modules\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344066 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344028    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-podres\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344229 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344079    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-proc\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344229 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344103    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-lib-modules\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344229 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344138    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lgv\" (UniqueName: \"kubernetes.io/projected/12a055f8-f29e-4c08-8f41-e11f693065ab-kube-api-access-v4lgv\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344229 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344216    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-podres\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344463 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344300    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-lib-modules\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344463 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344323    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-sys\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344463 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344399    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-sys\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.344463 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.344410    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12a055f8-f29e-4c08-8f41-e11f693065ab-proc\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.362330 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.362298    2584 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lgv\" (UniqueName: \"kubernetes.io/projected/12a055f8-f29e-4c08-8f41-e11f693065ab-kube-api-access-v4lgv\") pod \"perf-node-gather-daemonset-tz9vf\" (UID: \"12a055f8-f29e-4c08-8f41-e11f693065ab\") " pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.446250 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.446156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:41.733077 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:41.733008    2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"]
Sep 17 16:24:42.087345 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:42.087311    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf" event={"ID":"12a055f8-f29e-4c08-8f41-e11f693065ab","Type":"ContainerStarted","Data":"e8dd5602a46acd71efc9aa1938bd78298572313e404754faa2b0410237e52e98"}
Sep 17 16:24:42.087610 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:42.087587    2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf" event={"ID":"12a055f8-f29e-4c08-8f41-e11f693065ab","Type":"ContainerStarted","Data":"1291e250f81012cda9728520ae3660b8d499105ded8d7eb0354ee293027cce55"}
Sep 17 16:24:42.087764 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:42.087747    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:42.107606 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:42.107167    2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf" podStartSLOduration=1.107140924 podStartE2EDuration="1.107140924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:24:42.10522703 +0000 UTC m=+1840.331896343" watchObservedRunningTime="2026-09-17 16:24:42.107140924 +0000 UTC m=+1840.333810214"
Sep 17 16:24:42.632775 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:42.632748    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ptcg8_17dbdb6a-2110-41c9-b537-ba98feba5474/dns/0.log"
Sep 17 16:24:42.657749 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:42.657726    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ptcg8_17dbdb6a-2110-41c9-b537-ba98feba5474/kube-rbac-proxy/0.log"
Sep 17 16:24:42.684728 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:42.684696    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n4hh7_90b2d32e-fadd-4603-9036-138eefcb0411/dns-node-resolver/0.log"
Sep 17 16:24:43.341477 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:43.341449    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g82l6_fdbe84e1-dd8f-4701-93af-2087896382b3/node-ca/0.log"
Sep 17 16:24:44.541614 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:44.541584    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mwd2m_ff418b7a-d28f-45b6-a986-6855478a78c3/serve-healthcheck-canary/0.log"
Sep 17 16:24:45.031097 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:45.031058    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jclm2_8fd10f39-f258-4875-93f7-d9d4a0b1192d/kube-rbac-proxy/0.log"
Sep 17 16:24:45.053679 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:45.053649    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jclm2_8fd10f39-f258-4875-93f7-d9d4a0b1192d/exporter/0.log"
Sep 17 16:24:45.078104 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:45.078069    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jclm2_8fd10f39-f258-4875-93f7-d9d4a0b1192d/extractor/0.log"
Sep 17 16:24:47.573092 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:47.573056    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6fdd8844c6-t7rhk_5782e666-064b-4480-9e3a-1a38a1a3da63/manager/0.log"
Sep 17 16:24:47.631847 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:47.631819    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-d4sf4_76210352-393c-4666-8fa7-1e4930d11d1f/server/0.log"
Sep 17 16:24:47.836850 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:47.836732    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-tfcvd_960d461f-a29a-4ee3-b690-6ed3248c5203/manager/0.log"
Sep 17 16:24:49.105466 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:49.105434    2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vp8j5/perf-node-gather-daemonset-tz9vf"
Sep 17 16:24:54.222529 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.222500    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxc2j_00248c3f-bd71-4073-a6bb-de0a31df55f4/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:24:54.256672 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.256640    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxc2j_00248c3f-bd71-4073-a6bb-de0a31df55f4/egress-router-binary-copy/0.log"
Sep 17 16:24:54.281321 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.281291    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxc2j_00248c3f-bd71-4073-a6bb-de0a31df55f4/cni-plugins/0.log"
Sep 17 16:24:54.308793 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.308762    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxc2j_00248c3f-bd71-4073-a6bb-de0a31df55f4/bond-cni-plugin/0.log"
Sep 17 16:24:54.331503 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.331481    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxc2j_00248c3f-bd71-4073-a6bb-de0a31df55f4/routeoverride-cni/0.log"
Sep 17 16:24:54.355284 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.355259    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxc2j_00248c3f-bd71-4073-a6bb-de0a31df55f4/whereabouts-cni-bincopy/0.log"
Sep 17 16:24:54.378240 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.378199    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kxc2j_00248c3f-bd71-4073-a6bb-de0a31df55f4/whereabouts-cni/0.log"
Sep 17 16:24:54.455504 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.455468    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fqvct_d594aab7-c878-4797-b9a3-2e04a55a1586/kube-multus/0.log"
Sep 17 16:24:54.611257 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.611230    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wn4lc_b257fe13-9d6c-4f51-824f-3777dc25bff8/network-metrics-daemon/0.log"
Sep 17 16:24:54.639721 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:54.639687    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wn4lc_b257fe13-9d6c-4f51-824f-3777dc25bff8/kube-rbac-proxy/0.log"
Sep 17 16:24:55.651036 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:55.650988    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-szc22_3f913015-45d9-4ed6-9eae-931aaaefb6fd/ovn-controller/0.log"
Sep 17 16:24:55.687616 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:55.687586    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-szc22_3f913015-45d9-4ed6-9eae-931aaaefb6fd/ovn-acl-logging/0.log"
Sep 17 16:24:55.730531 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:55.730496    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-szc22_3f913015-45d9-4ed6-9eae-931aaaefb6fd/kube-rbac-proxy-node/0.log"
Sep 17 16:24:55.789264 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:55.789230    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-szc22_3f913015-45d9-4ed6-9eae-931aaaefb6fd/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 16:24:55.810266 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:55.810236    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-szc22_3f913015-45d9-4ed6-9eae-931aaaefb6fd/northd/0.log"
Sep 17 16:24:55.840057 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:55.840028    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-szc22_3f913015-45d9-4ed6-9eae-931aaaefb6fd/nbdb/0.log"
Sep 17 16:24:55.870584 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:55.870549    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-szc22_3f913015-45d9-4ed6-9eae-931aaaefb6fd/sbdb/0.log"
Sep 17 16:24:55.931617 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:55.931550    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-szc22_3f913015-45d9-4ed6-9eae-931aaaefb6fd/ovnkube-controller/0.log"
Sep 17 16:24:57.310693 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:57.310663    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-h9dg9_6fba78fa-721c-4986-ac7f-e7c994335e4b/network-check-target-container/0.log"
Sep 17 16:24:58.345913 ip-10-0-128-128 kubenswrapper[2584]: I0917 16:24:58.345882    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8bqtq_50a00b6e-f5c7-4f8a-88fe-8611d25ae671/iptables-alerter/0.log"