Sep 08 13:16:10.186012 ip-10-0-136-211 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:16:10.613018 ip-10-0-136-211 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:10.613018 ip-10-0-136-211 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:16:10.613018 ip-10-0-136-211 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:10.613018 ip-10-0-136-211 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:16:10.613018 ip-10-0-136-211 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:10.615230 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.615139    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:16:10.621908 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621889    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:10.621908 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621906    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:10.621908 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621910    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621914    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621917    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621920    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621923    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621926    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621928    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621931    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621933    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621936    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621938    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621941    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621944    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621946    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621949    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621952    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621955    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621958    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621960    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621963    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:10.622013 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621966    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621968    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621971    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621973    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621976    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621979    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621982    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621984    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621987    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621989    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621992    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621994    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621997    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.621999    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622003    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622006    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622008    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622011    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622014    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622017    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:10.622497 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622020    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622023    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622025    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622028    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622031    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622033    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622036    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622038    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622041    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622044    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622046    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622049    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622052    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622054    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622063    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622066    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622068    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622071    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622074    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622077    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:10.623388 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622079    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622082    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622084    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622089    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622092    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622095    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622099    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622103    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622105    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622108    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622111    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622113    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622116    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622119    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622121    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622124    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622126    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622128    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622131    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622134    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:10.624043 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622137    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622140    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622143    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622145    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622148    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622150    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622153    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622155    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622159    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622162    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622166    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622169    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622172    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622175    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622178    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622180    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:10.624547 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.622183    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624570    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624583    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624586    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624589    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624592    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624595    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624598    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624601    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624603    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624606    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624608    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624611    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624616    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624620    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624624    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624628    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624630    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624633    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624636    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:10.624971 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624639    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624641    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624644    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624647    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624649    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624652    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624654    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624657    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624660    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624663    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624666    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624684    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624687    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624690    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624693    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624696    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624698    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624701    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624703    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624706    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:10.625493 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624709    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624711    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624714    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624716    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624719    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624722    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624724    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624726    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624729    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624731    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624734    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624736    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624739    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624741    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624744    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624746    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624749    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624752    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624755    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624759    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:10.626032 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624763    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624765    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624768    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624770    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624773    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624776    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624779    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624782    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624785    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624787    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624790    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624792    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624795    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624797    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624799    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624803    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624805    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624808    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624810    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624813    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:10.626530 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624816    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624818    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624821    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624823    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624826    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624829    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624831    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624834    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624836    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624839    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624841    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624844    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624846    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624848    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624851    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624853    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624856    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624859    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624865    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:10.627042 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.624867    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625302    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625309    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625312    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625316    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625320    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625323    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625326    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625329    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625332    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625334    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625337    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625339    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625342    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625344    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625347    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625349    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625352    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625355    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:10.627523 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625358    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625360    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625362    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625365    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625368    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625371    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625373    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625376    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625378    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625381    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625383    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625386    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625390    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625392    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625395    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625397    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625400    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625404    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625407    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625409    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:10.628007 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625412    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625414    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625417    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625419    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625422    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625424    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625426    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625429    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625431    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625434    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625436    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625440    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625442    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625445    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625447    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625450    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625453    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625455    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625458    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625460    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:10.628506 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625463    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625465    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625468    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625471    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625474    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625476    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625479    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625481    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625484    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625487    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625489    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625492    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625495    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625497    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625500    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625503    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625506    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625509    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625511    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625514    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:10.629017 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625516    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625519    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625521    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625523    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625527    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625529    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625533    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625535    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625537    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625540    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625542    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625545    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625547    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625549    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625552    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625556    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625560    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625563    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625566    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:10.629499 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625569    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.625572    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625655    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625715    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625724    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625728    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625733    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625736    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625741    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625745    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625748    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625751    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625755    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625758    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625761    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625764    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625767    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625769    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625773    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625776    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625780    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625783    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625786    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:16:10.629980 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625789    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625792    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625796    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625799    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625802    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625805    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625808    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625811    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625814    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625817    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625838    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625843    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625846    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625849    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625852    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625855    2581 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625858    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625863    2581 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625866    2581 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625868    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625871    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625874    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625878    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625880    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625883    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:16:10.630528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625886    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625889    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625891    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625896    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625899    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625902    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625905    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625907    2581 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625911    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625914    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625917    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625920    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625923    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625928    2581 flags.go:64] FLAG: --help="false"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625931    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-136-211.ec2.internal"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625934    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625937    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625939    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625943    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625946    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625949    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625952    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625954    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:16:10.631150 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625957    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625960    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625963    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625965    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625968    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625971    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625974    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625976    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625979    2581 flags.go:64] FLAG: --lock-file=""
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625982    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625985    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625988    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625993    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625997    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.625999    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626002    2581 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626005    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626008    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626011    2581 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626014    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626019    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626021    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626029    2581 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626032    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626035    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:16:10.631789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626038    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626041    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626044    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626047    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626050    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626057    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626060    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626063    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626066    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626069    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626075    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626077    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626080    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626083    2581 flags.go:64] FLAG: --port="10250"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626086    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626089    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04dffcdd4a170ec80"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626092    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626095    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626098    2581 flags.go:64] FLAG: --register-node="true"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626101    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626104    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626109    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626112    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626115    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626118    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:16:10.632415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626121    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626125    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626128    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626131    2581 flags.go:64] FLAG: --runonce="false"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626134    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626138    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626141    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626144    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626147    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626150    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626153    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626156    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626158    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626161    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626164    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626167    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626170    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626173    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626176    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626181    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626183    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626186    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626190    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626193    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626195    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:16:10.633050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626198    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626201    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626204    2581 flags.go:64] FLAG: --v="2"
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626208    2581 flags.go:64] FLAG: --version="false"
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626213    2581 flags.go:64] FLAG: --vmodule=""
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626216    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.626220    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626365    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626370    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626373    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626376    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626379    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626384    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626387    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626389    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626392    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626394    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626397    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626399    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626402    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626404    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:10.633681 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626407    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626410    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626413    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626415    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626418    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626420    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626423    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626425    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626429    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626431    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626433    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626436    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626438    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626441    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626443    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626446    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626449    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626451    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626454    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626458    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626460    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:10.634205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626462    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626465    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626468    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626471    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626473    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626476    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626478    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626481    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626483    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626485    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626488    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626491    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626493    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626498    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626501    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626504    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626506    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626509    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626511    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:10.634737 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626514    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626517    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626519    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626522    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626525    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626529    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626531    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626534    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626537    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626540    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626543    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626547    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626550    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626552    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626554    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626559    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626562    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626564    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626566    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:10.635242 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626569    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626572    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626574    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626576    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626579    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626582    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626584    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626587    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626589    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626592    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626594    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626597    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626599    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626602    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626604    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626607    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626609    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626614    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626617    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:10.635744 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626631    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626634    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626637    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626640    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626643    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626647    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.626650    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.627547    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.635297    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:16:10.636216 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.635316    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.636227    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.636234    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636298    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636304    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636307    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636311    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636314    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636317    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636319    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636322    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636325    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636328    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636330    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636333    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636336    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636338    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636341    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636343    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636346    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:10.636455 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636349    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636351    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636354    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636356    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636359    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636361    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636364    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636366    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636369    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636371    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636373    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636376    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636380    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636382    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636388    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636392    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636395    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636398    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636401    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:10.636963 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636404    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636407    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636411    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636414    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636417    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636420    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636424    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636427    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636430    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636432    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636435    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636438    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636440    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636443    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636445    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636447    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636450    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636453    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636455    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:10.637444 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636458    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636460    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636463    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636466    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636468    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636471    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636473    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636476    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636480    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636483    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636485    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636488    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636490    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636493    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636495    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636498    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636500    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636502    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636505    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:10.637919 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636507    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636510    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636512    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636515    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636517    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636519    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636522    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636524    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636527    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636529    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636531    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636534    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636536    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636539    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636541    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636544    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636546    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636549    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636551    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636553    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:10.638417 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636556    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636560    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636563    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636565    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636567    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.636573    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636707    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636712    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636715    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636718    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636720    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636723    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636725    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636728    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636730    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:10.638929 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636733    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636735    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636738    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636740    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636743    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636746    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636748    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636751    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636753    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636756    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636758    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636761    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636763    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636766    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636768    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636771    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636773    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636776    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636779    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636782    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:10.639305 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636784    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636788    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636792    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636795    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636798    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636801    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636803    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636806    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636808    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636811    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636814    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636816    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636818    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636821    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636823    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636826    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636828    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636831    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636834    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636836    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:10.639843 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636839    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636841    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636844    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636846    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636849    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636851    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636854    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636856    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636860    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636863    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636866    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636869    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636872    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636875    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636878    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636881    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636883    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636885    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636888    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:10.640331 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636891    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636893    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636896    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636898    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636900    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636903    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636905    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636908    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636911    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636913    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636915    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636918    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636920    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636923    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636925    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636927    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636930    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636932    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636935    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:10.640823 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636937    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636940    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636942    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636945    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636948    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636954    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636957    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636961    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636963    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636966    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636969    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:10.636972    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.636977    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.637102    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.639561    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:16:10.641273 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.640598    2581 server.go:1019] "Starting client certificate rotation"
Sep 08 13:16:10.641641 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.640706    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:10.641641 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.640740    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:10.643447 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.643433    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:10.645712 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.645693    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:10.655715 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.655694    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:16:10.659089 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.659070    2581 log.go:25] "Validated CRI v1 image API"
Sep 08 13:16:10.659964 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.659946    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:16:10.665071 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.665051    2581 fs.go:135] Filesystem UUIDs: map[23de3b90-2ea5-4fc2-8763-785b9abdea17:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7ea98f4e-a416-4b54-9560-ccacce875218:/dev/nvme0n1p3]
Sep 08 13:16:10.665142 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.665070    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:16:10.671158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.671047    2581 manager.go:216] Machine: {Timestamp:2026-09-08 13:16:10.669688108 +0000 UTC m=+0.373473947 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093980 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ce3cfa6e0edf1e609efe96879aceb SystemUUID:ec2ce3cf-a6e0-edf1-e609-efe96879aceb BootID:14078a35-a7fa-404c-b637-2fff41b3af4d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e5:4f:9a:3e:1b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e5:4f:9a:3e:1b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:e8:c6:80:20:23 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:16:10.671158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.671150    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:16:10.671281 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.671228    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:16:10.674706 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.674658    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:16:10.674833 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.674820    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:10.674864 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.674711    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-211.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:16:10.675644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.675634    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:16:10.675690 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.675647    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:16:10.675690 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.675655    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:10.675690 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.675682    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:10.676950 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.676939    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:16:10.681705 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.681694    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:10.681812 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.681803    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:16:10.681855 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.681838    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:16:10.681884 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.681855    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:16:10.681884 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.681871    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:16:10.681884 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.681880    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:16:10.681991 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.681889    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:16:10.682256 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.682236    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:10.682295 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.682264    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:10.684550 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.684531    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:16:10.687536 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.687517    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:16:10.687622 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.687557    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:16:10.689705 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689692    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:16:10.689786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689714    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:16:10.689786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689728    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:16:10.689786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689736    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:16:10.689786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689747    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:16:10.689786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689755    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:16:10.689786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689778    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:16:10.689786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689786    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:16:10.690029 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689798    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:16:10.690029 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689806    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:16:10.690029 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689830    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:16:10.690029 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.689840    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:16:10.690808 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.690796    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:16:10.690863 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.690811    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:16:10.692582 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.692553    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:16:10.692959 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.692720    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:16:10.693039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.693015    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-211.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:16:10.695349 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.695331    2581 server.go:1268] "Started kubelet"
Sep 08 13:16:10.695436 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.695396    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:16:10.695471 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.695430    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:16:10.695517 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.695482    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:16:10.696249 ip-10-0-136-211 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:16:10.697096 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.697005    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:16:10.697481 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.697451    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:16:10.700686 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.700653    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:16:10.704506 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.704489    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:10.705216 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.705205    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:16:10.705296 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.705275    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:10.706234 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706121    2581 factory.go:55] Registering systemd factory
Sep 08 13:16:10.706234 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706159    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:16:10.706324 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706239    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:16:10.706393 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706379    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:16:10.706469 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706460    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:16:10.706574 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706554    2581 factory.go:153] Registering CRI-O factory
Sep 08 13:16:10.706634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706578    2581 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:16:10.706634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706616    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:16:10.706634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706626    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:16:10.706634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706630    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:16:10.707008 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706652    2581 factory.go:103] Registering Raw factory
Sep 08 13:16:10.707008 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.706690    2581 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:16:10.707109 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.707024    2581 manager.go:349] Starting recovery of all containers
Sep 08 13:16:10.710798 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.710766    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-211.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:16:10.710798 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.710784    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:16:10.712046 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.710865    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-211.ec2.internal.18d35ada5072eb70  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-211.ec2.internal,UID:ip-10-0-136-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-211.ec2.internal,},FirstTimestamp:2026-09-08 13:16:10.695297904 +0000 UTC m=+0.399083744,LastTimestamp:2026-09-08 13:16:10.695297904 +0000 UTC m=+0.399083744,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-211.ec2.internal,}"
Sep 08 13:16:10.717718 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.717697    2581 manager.go:354] Recovery completed
Sep 08 13:16:10.722415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.722402    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:10.722915 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.722899    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n2nkf"
Sep 08 13:16:10.725447 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.725433    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:10.725512 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.725462    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:10.725512 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.725472    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:10.726018 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.726001    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:16:10.726018 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.726013    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:16:10.726018 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.726016    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:10.726146 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.726027    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:10.726781 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.726766    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:10.726823 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.726794    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:10.726823 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.726804    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:10.727599 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.727540    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-211.ec2.internal.18d35ada523efb72  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-211.ec2.internal,UID:ip-10-0-136-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-211.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-211.ec2.internal,},FirstTimestamp:2026-09-08 13:16:10.725448562 +0000 UTC m=+0.429234402,LastTimestamp:2026-09-08 13:16:10.725448562 +0000 UTC m=+0.429234402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-211.ec2.internal,}"
Sep 08 13:16:10.728188 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.728177    2581 policy_none.go:49] "None policy: Start"
Sep 08 13:16:10.728228 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.728192    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:16:10.728228 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.728201    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:16:10.730187 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.730173    2581 policy_none.go:47] "Start"
Sep 08 13:16:10.734538 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.734479    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-211.ec2.internal.18d35ada523f4209  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-211.ec2.internal,UID:ip-10-0-136-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-211.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-211.ec2.internal,},FirstTimestamp:2026-09-08 13:16:10.725466633 +0000 UTC m=+0.429252473,LastTimestamp:2026-09-08 13:16:10.725466633 +0000 UTC m=+0.429252473,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-211.ec2.internal,}"
Sep 08 13:16:10.735301 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.735283    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n2nkf"
Sep 08 13:16:10.738300 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.738239    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-211.ec2.internal.18d35ada523f6597  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-211.ec2.internal,UID:ip-10-0-136-211.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-211.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-211.ec2.internal,},FirstTimestamp:2026-09-08 13:16:10.725475735 +0000 UTC m=+0.429261578,LastTimestamp:2026-09-08 13:16:10.725475735 +0000 UTC m=+0.429261578,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-211.ec2.internal,}"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.773183    2581 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.773216    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.773226    2581 server.go:87] "Starting device plugin registration server"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.773428    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.773440    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.773581    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.773657    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.773665    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.774205    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:16:10.791891 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.774249    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:10.815221 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.815184    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:16:10.816543 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.816520    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:16:10.816631 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.816568    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:16:10.816631 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.816592    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:16:10.816734 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.816632    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:16:10.818600 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.818578    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:10.873879 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.873791    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:10.874858 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.874842    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:10.874937 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.874875    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:10.874937 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.874885    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:10.874937 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.874907    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-211.ec2.internal"
Sep 08 13:16:10.883086 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.883070    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-211.ec2.internal"
Sep 08 13:16:10.883136 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.883095    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-211.ec2.internal\": node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:10.916954 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.916920    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal"]
Sep 08 13:16:10.917134 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.917007    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:10.917134 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.917102    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:10.917990 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.917964    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:10.918107 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.917998    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:10.918107 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.918008    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:10.920310 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.920288    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:10.920447 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.920433    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:10.920485 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.920462    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:10.921049 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.921033    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:10.921135 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.921048    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:10.921135 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.921058    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:10.921135 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.921067    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:10.921135 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.921072    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:10.921135 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.921076    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:10.923273 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.923256    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:10.923360 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.923284    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:10.923985 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.923971    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:10.924085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.923994    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:10.924085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:10.924007    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:10.957256 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.957239    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-211.ec2.internal\" not found" node="ip-10-0-136-211.ec2.internal"
Sep 08 13:16:10.961582 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:10.961564    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-211.ec2.internal\" not found" node="ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.008461 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.008435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e342b94596d41ae05df25539ced6fa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal\" (UID: \"2e342b94596d41ae05df25539ced6fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.008461 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.008463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4084ecab986bde65b84b8c14d785fe7d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-211.ec2.internal\" (UID: \"4084ecab986bde65b84b8c14d785fe7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.008628 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.008479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e342b94596d41ae05df25539ced6fa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal\" (UID: \"2e342b94596d41ae05df25539ced6fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.017651 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.017631    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.109156 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.109120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4084ecab986bde65b84b8c14d785fe7d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-211.ec2.internal\" (UID: \"4084ecab986bde65b84b8c14d785fe7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.109156 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.109159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e342b94596d41ae05df25539ced6fa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal\" (UID: \"2e342b94596d41ae05df25539ced6fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.109290 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.109180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e342b94596d41ae05df25539ced6fa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal\" (UID: \"2e342b94596d41ae05df25539ced6fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.109290 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.109210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e342b94596d41ae05df25539ced6fa9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal\" (UID: \"2e342b94596d41ae05df25539ced6fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.109290 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.109231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e342b94596d41ae05df25539ced6fa9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal\" (UID: \"2e342b94596d41ae05df25539ced6fa9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.109290 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.109210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4084ecab986bde65b84b8c14d785fe7d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-211.ec2.internal\" (UID: \"4084ecab986bde65b84b8c14d785fe7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.118244 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.118217    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.219064 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.218995    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.259175 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.259150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.265140 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.265124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:11.320238 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.320061    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.420581 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.420553    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.521194 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.521141    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.621757 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.621724    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.640894 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.640874    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:16:11.641009 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.640993    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:11.675906 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.675884    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:11.708511 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.708490    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:11.722386 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.722360    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.741534 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.741500    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:11:10 +0000 UTC" deadline="2028-03-17 17:40:24.278320855 +0000 UTC"
Sep 08 13:16:11.741534 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.741531    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13348h24m12.536792117s"
Sep 08 13:16:11.823084 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.823058    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:11.823194 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:11.823092    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4084ecab986bde65b84b8c14d785fe7d.slice/crio-b781c336258e6e677428a66f1af7d3d3fa036953196db37f89561a925701555f WatchSource:0}: Error finding container b781c336258e6e677428a66f1af7d3d3fa036953196db37f89561a925701555f: Status 404 returned error can't find the container with id b781c336258e6e677428a66f1af7d3d3fa036953196db37f89561a925701555f
Sep 08 13:16:11.827568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.827553    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:16:11.914209 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.914174    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:11.915270 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.915247    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:11.915394 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.915284    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:11.915394 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:11.915300    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:11.923128 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:11.923110    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:12.023802 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.023762    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-211.ec2.internal\" not found"
Sep 08 13:16:12.062064 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.062002    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:12.106473 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.106437    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:12.118206 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.118180    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:12.119244 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.119228    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"
Sep 08 13:16:12.132133 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.132112    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:12.634688 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.634659    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:12.684803 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.684770    2581 apiserver.go:52] "Watching apiserver"
Sep 08 13:16:12.690987 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.690965    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:16:12.692422 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.692398    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-sgl7l","openshift-network-diagnostics/network-check-target-f7wdv","openshift-network-operator/iptables-alerter-m7j8b","openshift-cluster-node-tuning-operator/tuned-59gzp","openshift-image-registry/node-ca-lnvw8","openshift-multus/multus-g9wzq","openshift-multus/network-metrics-daemon-mpqn2","openshift-ovn-kubernetes/ovnkube-node-c2n4x","kube-system/konnectivity-agent-wnrbm","kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9","openshift-dns/node-resolver-bxllr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal"]
Sep 08 13:16:12.696249 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.696226    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.698551 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.698527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:12.698663 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.698549    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:12.698663 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.698579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:12.698663 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.698595    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:16:12.698663 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.698607    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:12.698925 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.698902    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sdzsd\""
Sep 08 13:16:12.700900 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.700882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:12.700985 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.700942    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:12.703163 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.703145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.705224 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.705207    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:12.705596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.705564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.706123 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.706103    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:12.706398 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.706383    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kqxw\""
Sep 08 13:16:12.706526 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.706511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:12.707993 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.707967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.710374 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.710359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.713026 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.713012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.714476 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.714129    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:16:12.714476 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.714332    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:16:12.714619 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.714329    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:16:12.714671 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.714615    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:16:12.715086 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.715065    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.715645    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.715773    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.715860    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6svj4\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.716038    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.716086    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.716087    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6ggnj\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.716238    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zw9kn\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.716294    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vvd4m\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.716820    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717049    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717058    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717128    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:16:12.717644 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717409    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-system-cni-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-k8s-cni-cncf-io\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-cni-bin\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-kubernetes\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.717986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-tmp\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b804ad43-3ef5-4847-914e-456974ad6d9b-cni-binary-copy\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-daemon-config\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-multus-certs\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-lib-modules\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.718680 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718360    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g864g\" (UniqueName: \"kubernetes.io/projected/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-kube-api-access-g864g\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.718680 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-conf-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718680 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72461732-cf32-4137-a68c-016f7d5d785a-host-slash\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.718680 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-run\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.718680 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-kubelet\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718680 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4fl\" (UniqueName: \"kubernetes.io/projected/b804ad43-3ef5-4847-914e-456974ad6d9b-kube-api-access-zw4fl\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.718952 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-host\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.718952 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-tuned\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.718952 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.718767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-cni-multus\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.719166 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98rs\" (UniqueName: \"kubernetes.io/projected/72461732-cf32-4137-a68c-016f7d5d785a-kube-api-access-h98rs\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.719212 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysctl-conf\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.719212 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-host\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.719212 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-cnibin\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.719385 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72461732-cf32-4137-a68c-016f7d5d785a-iptables-alerter-script\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.719385 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:12.719385 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-systemd\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.719547 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-sys\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.719547 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719480    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-serviceca\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.719547 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wb8l\" (UniqueName: \"kubernetes.io/projected/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-kube-api-access-2wb8l\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.719547 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-netns\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.719774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719560    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-hostroot\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.719774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-modprobe-d\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.719774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysconfig\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.719774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysctl-d\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.719774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-var-lib-kubelet\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.719774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-cni-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.719774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-socket-dir-parent\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.720117 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-etc-kubernetes\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.720117 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719811    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:12.720117 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87rtd\" (UniqueName: \"kubernetes.io/projected/1a5ec803-ab0d-4166-bf48-77a72a26b284-kube-api-access-87rtd\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:12.720117 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719860    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:16:12.720117 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.719995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:16:12.721027 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.721009    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:16:12.721116 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.721065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-x55z2\""
Sep 08 13:16:12.721537 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.721522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.723978 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.723945    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2qng6\""
Sep 08 13:16:12.723978 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.723967    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:16:12.724144 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.723989    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.724144 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.724056    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:16:12.724144 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.724113    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:16:12.727954 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.727940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vwvxm\""
Sep 08 13:16:12.728291 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.728275    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:16:12.728338 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.728280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:16:12.728738 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.728713    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:12.757311 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.757287    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-c4vlg"
Sep 08 13:16:12.764610 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.764593    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-c4vlg"
Sep 08 13:16:12.807577 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.807552    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:16:12.820094 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820060    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-cni-multus\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.820094 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cnibin\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.820282 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.820282 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovnkube-config\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.820282 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820180    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-cni-multus\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.820282 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovn-node-metrics-cert\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.820282 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f80b4d3-387c-4580-9899-0be849014b1e-tmp-dir\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67d5\" (UniqueName: \"kubernetes.io/projected/2f80b4d3-387c-4580-9899-0be849014b1e-kube-api-access-k67d5\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-host\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-cnibin\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-systemd\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-sys\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-host\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-cnibin\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-systemd-units\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.820542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-sys\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820544    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-systemd\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvxw4\" (UniqueName: \"kubernetes.io/projected/0c2245f7-7abe-45bd-8ab1-d500057643ff-kube-api-access-pvxw4\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-serviceca\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-netns\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820664    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-netns\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-modprobe-d\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysconfig\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysctl-d\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-modprobe-d\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vn26\" (UniqueName: \"kubernetes.io/projected/d077d15f-0206-485e-a148-5d4c0a3e9ccf-kube-api-access-8vn26\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-cni-netd\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-socket-dir-parent\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.820994    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-device-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysconfig\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7e1ca70-26c8-46d3-ad76-7bea5f11a16c-agent-certs\") pod \"konnectivity-agent-wnrbm\" (UID: \"c7e1ca70-26c8-46d3-ad76-7bea5f11a16c\") " pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysctl-d\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-kubernetes\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-socket-dir-parent\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821203    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-kubernetes\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-multus-certs\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-serviceca\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-multus-certs\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-lib-modules\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovnkube-script-lib\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-lib-modules\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-conf-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72461732-cf32-4137-a68c-016f7d5d785a-host-slash\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-run\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-conf-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-ovn\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-kubelet\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4fl\" (UniqueName: \"kubernetes.io/projected/b804ad43-3ef5-4847-914e-456974ad6d9b-kube-api-access-zw4fl\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.821809 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72461732-cf32-4137-a68c-016f7d5d785a-host-slash\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821886    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-run\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-host\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.821982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-kubelet\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822006    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-systemd\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822075    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-host\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98rs\" (UniqueName: \"kubernetes.io/projected/72461732-cf32-4137-a68c-016f7d5d785a-kube-api-access-h98rs\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysctl-conf\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-slash\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822170    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822194    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72461732-cf32-4137-a68c-016f7d5d785a-iptables-alerter-script\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-node-log\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f80b4d3-387c-4580-9899-0be849014b1e-hosts-file\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wb8l\" (UniqueName: \"kubernetes.io/projected/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-kube-api-access-2wb8l\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-hostroot\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.822483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-var-lib-kubelet\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal" event={"ID":"4084ecab986bde65b84b8c14d785fe7d","Type":"ContainerStarted","Data":"b781c336258e6e677428a66f1af7d3d3fa036953196db37f89561a925701555f"}
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-sysctl-conf\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-run-netns\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-hostroot\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-etc-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-var-lib-kubelet\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7e1ca70-26c8-46d3-ad76-7bea5f11a16c-konnectivity-ca\") pod \"konnectivity-agent-wnrbm\" (UID: \"c7e1ca70-26c8-46d3-ad76-7bea5f11a16c\") " pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-cni-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-etc-kubernetes\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87rtd\" (UniqueName: \"kubernetes.io/projected/1a5ec803-ab0d-4166-bf48-77a72a26b284-kube-api-access-87rtd\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-cni-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-env-overrides\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72461732-cf32-4137-a68c-016f7d5d785a-iptables-alerter-script\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.823006 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-system-cni-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-k8s-cni-cncf-io\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.822916    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-cni-bin\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-tmp\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsr5z\" (UniqueName: \"kubernetes.io/projected/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-kube-api-access-bsr5z\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-log-socket\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.823110    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:13.323084406 +0000 UTC m=+3.026870248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-run-k8s-cni-cncf-io\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.822897    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-system-cni-dir\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-etc-kubernetes\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b804ad43-3ef5-4847-914e-456974ad6d9b-host-var-lib-cni-bin\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b804ad43-3ef5-4847-914e-456974ad6d9b-cni-binary-copy\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-daemon-config\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g864g\" (UniqueName: \"kubernetes.io/projected/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-kube-api-access-g864g\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.823568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823491    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-registration-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-sys-fs\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-socket-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-var-lib-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-cni-bin\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-tuned\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-system-cni-dir\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.823968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-kubelet\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.824001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.824329 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.824137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b804ad43-3ef5-4847-914e-456974ad6d9b-cni-binary-copy\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.825461 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.825405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b804ad43-3ef5-4847-914e-456974ad6d9b-multus-daemon-config\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.827652 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.827631    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-tmp\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.829201 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.829178    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-etc-tuned\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.833032 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.833013    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:12.833114 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.833037    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:12.833114 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.833049    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hzsss for pod openshift-network-diagnostics/network-check-target-f7wdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:12.833182 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:12.833119    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss podName:cd92145f-03d4-44bc-ab70-24feb2c337f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:13.333104395 +0000 UTC m=+3.036890224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hzsss" (UniqueName: "kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss") pod "network-check-target-f7wdv" (UID: "cd92145f-03d4-44bc-ab70-24feb2c337f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:12.836812 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.836785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4fl\" (UniqueName: \"kubernetes.io/projected/b804ad43-3ef5-4847-914e-456974ad6d9b-kube-api-access-zw4fl\") pod \"multus-g9wzq\" (UID: \"b804ad43-3ef5-4847-914e-456974ad6d9b\") " pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:12.839107 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.839080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wb8l\" (UniqueName: \"kubernetes.io/projected/d3fd4abc-ff7d-4512-a9f2-8103510ca5c6-kube-api-access-2wb8l\") pod \"node-ca-lnvw8\" (UID: \"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6\") " pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:12.839836 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.839816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98rs\" (UniqueName: \"kubernetes.io/projected/72461732-cf32-4137-a68c-016f7d5d785a-kube-api-access-h98rs\") pod \"iptables-alerter-m7j8b\" (UID: \"72461732-cf32-4137-a68c-016f7d5d785a\") " pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:12.839956 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.839938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g864g\" (UniqueName: \"kubernetes.io/projected/fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2-kube-api-access-g864g\") pod \"tuned-59gzp\" (UID: \"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2\") " pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:12.846120 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.846100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87rtd\" (UniqueName: \"kubernetes.io/projected/1a5ec803-ab0d-4166-bf48-77a72a26b284-kube-api-access-87rtd\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:12.925090 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.924998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-env-overrides\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.925090 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.925090 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsr5z\" (UniqueName: \"kubernetes.io/projected/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-kube-api-access-bsr5z\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.925090 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-log-socket\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-registration-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-sys-fs\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-socket-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-var-lib-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-cni-bin\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-system-cni-dir\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-sys-fs\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-registration-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-kubelet\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-log-socket\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925328    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-var-lib-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-socket-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-system-cni-dir\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.925377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-cni-bin\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-kubelet\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-etc-selinux\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cnibin\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cnibin\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovnkube-config\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovn-node-metrics-cert\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f80b4d3-387c-4580-9899-0be849014b1e-tmp-dir\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67d5\" (UniqueName: \"kubernetes.io/projected/2f80b4d3-387c-4580-9899-0be849014b1e-kube-api-access-k67d5\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-systemd-units\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvxw4\" (UniqueName: \"kubernetes.io/projected/0c2245f7-7abe-45bd-8ab1-d500057643ff-kube-api-access-pvxw4\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vn26\" (UniqueName: \"kubernetes.io/projected/d077d15f-0206-485e-a148-5d4c0a3e9ccf-kube-api-access-8vn26\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-systemd-units\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-cni-netd\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-device-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7e1ca70-26c8-46d3-ad76-7bea5f11a16c-agent-certs\") pod \"konnectivity-agent-wnrbm\" (UID: \"c7e1ca70-26c8-46d3-ad76-7bea5f11a16c\") " pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovnkube-script-lib\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-ovn\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.925969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f80b4d3-387c-4580-9899-0be849014b1e-tmp-dir\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-systemd\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-cni-netd\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-slash\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-device-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-node-log\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f80b4d3-387c-4580-9899-0be849014b1e-hosts-file\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.926969 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-ovn\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-run-netns\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-env-overrides\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-etc-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-slash\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926312    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7e1ca70-26c8-46d3-ad76-7bea5f11a16c-konnectivity-ca\") pod \"konnectivity-agent-wnrbm\" (UID: \"c7e1ca70-26c8-46d3-ad76-7bea5f11a16c\") " pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-systemd\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d077d15f-0206-485e-a148-5d4c0a3e9ccf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovnkube-config\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-run-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926486    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-host-run-netns\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926524    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-etc-openvswitch\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0c2245f7-7abe-45bd-8ab1-d500057643ff-node-log\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2f80b4d3-387c-4580-9899-0be849014b1e-hosts-file\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovnkube-script-lib\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.927452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-cni-binary-copy\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.927982 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.926945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7e1ca70-26c8-46d3-ad76-7bea5f11a16c-konnectivity-ca\") pod \"konnectivity-agent-wnrbm\" (UID: \"c7e1ca70-26c8-46d3-ad76-7bea5f11a16c\") " pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:12.928611 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.928591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7e1ca70-26c8-46d3-ad76-7bea5f11a16c-agent-certs\") pod \"konnectivity-agent-wnrbm\" (UID: \"c7e1ca70-26c8-46d3-ad76-7bea5f11a16c\") " pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:12.928687 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.928656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0c2245f7-7abe-45bd-8ab1-d500057643ff-ovn-node-metrics-cert\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:12.933951 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.933931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vn26\" (UniqueName: \"kubernetes.io/projected/d077d15f-0206-485e-a148-5d4c0a3e9ccf-kube-api-access-8vn26\") pod \"aws-ebs-csi-driver-node-dbcm9\" (UID: \"d077d15f-0206-485e-a148-5d4c0a3e9ccf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:12.934250 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.934230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsr5z\" (UniqueName: \"kubernetes.io/projected/93bed6de-bafb-4a04-aa8b-2a4f94ff1497-kube-api-access-bsr5z\") pod \"multus-additional-cni-plugins-sgl7l\" (UID: \"93bed6de-bafb-4a04-aa8b-2a4f94ff1497\") " pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:12.934288 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.934240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67d5\" (UniqueName: \"kubernetes.io/projected/2f80b4d3-387c-4580-9899-0be849014b1e-kube-api-access-k67d5\") pod \"node-resolver-bxllr\" (UID: \"2f80b4d3-387c-4580-9899-0be849014b1e\") " pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:12.935392 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:12.935376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvxw4\" (UniqueName: \"kubernetes.io/projected/0c2245f7-7abe-45bd-8ab1-d500057643ff-kube-api-access-pvxw4\") pod \"ovnkube-node-c2n4x\" (UID: \"0c2245f7-7abe-45bd-8ab1-d500057643ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:13.007848 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.007780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m7j8b"
Sep 08 13:16:13.015128 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.015100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-59gzp"
Sep 08 13:16:13.019390 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:13.019363    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72461732_cf32_4137_a68c_016f7d5d785a.slice/crio-733727f9020e01541715337d9296e4d9394123c2fa4b7b60004757325b0cd447 WatchSource:0}: Error finding container 733727f9020e01541715337d9296e4d9394123c2fa4b7b60004757325b0cd447: Status 404 returned error can't find the container with id 733727f9020e01541715337d9296e4d9394123c2fa4b7b60004757325b0cd447
Sep 08 13:16:13.024618 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.024587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lnvw8"
Sep 08 13:16:13.024973 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:13.024952    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8ad5d7_0e36_4ab4_b6e3_858afa8715b2.slice/crio-a52067b5141ef8689dfb17892ad3131c6ebe94585d576b66408643e332713c4f WatchSource:0}: Error finding container a52067b5141ef8689dfb17892ad3131c6ebe94585d576b66408643e332713c4f: Status 404 returned error can't find the container with id a52067b5141ef8689dfb17892ad3131c6ebe94585d576b66408643e332713c4f
Sep 08 13:16:13.029564 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.029540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g9wzq"
Sep 08 13:16:13.034297 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:13.034272    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fd4abc_ff7d_4512_a9f2_8103510ca5c6.slice/crio-c349053a0780acd7b8bbeb1325836f2e00ef200279c5937a4e2ddb97477de159 WatchSource:0}: Error finding container c349053a0780acd7b8bbeb1325836f2e00ef200279c5937a4e2ddb97477de159: Status 404 returned error can't find the container with id c349053a0780acd7b8bbeb1325836f2e00ef200279c5937a4e2ddb97477de159
Sep 08 13:16:13.035256 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.035221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sgl7l"
Sep 08 13:16:13.039715 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:13.039660    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb804ad43_3ef5_4847_914e_456974ad6d9b.slice/crio-a7c2d8ebb0ebced2268121b85ca3812160f72be6944e3836a4ef7311d8af9026 WatchSource:0}: Error finding container a7c2d8ebb0ebced2268121b85ca3812160f72be6944e3836a4ef7311d8af9026: Status 404 returned error can't find the container with id a7c2d8ebb0ebced2268121b85ca3812160f72be6944e3836a4ef7311d8af9026
Sep 08 13:16:13.040951 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.040929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:13.045665 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:13.045642    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bed6de_bafb_4a04_aa8b_2a4f94ff1497.slice/crio-b8c707f49673a160301be2b8cd0b38d540aa0d4a5aaaa121312becf6a1363875 WatchSource:0}: Error finding container b8c707f49673a160301be2b8cd0b38d540aa0d4a5aaaa121312becf6a1363875: Status 404 returned error can't find the container with id b8c707f49673a160301be2b8cd0b38d540aa0d4a5aaaa121312becf6a1363875
Sep 08 13:16:13.046223 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.046203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:13.050324 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:13.050303    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2245f7_7abe_45bd_8ab1_d500057643ff.slice/crio-e56960d4ec1624f3130633ba1a39a1c07e88e1f32196ffcc351b77722b4d8494 WatchSource:0}: Error finding container e56960d4ec1624f3130633ba1a39a1c07e88e1f32196ffcc351b77722b4d8494: Status 404 returned error can't find the container with id e56960d4ec1624f3130633ba1a39a1c07e88e1f32196ffcc351b77722b4d8494
Sep 08 13:16:13.051975 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.051759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9"
Sep 08 13:16:13.056534 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.056508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bxllr"
Sep 08 13:16:13.064790 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:13.064762    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd077d15f_0206_485e_a148_5d4c0a3e9ccf.slice/crio-53ae8925b132dd02a1e2cbfa688bd6cb410e7fd56266e6cd7d1c7d6041e1a264 WatchSource:0}: Error finding container 53ae8925b132dd02a1e2cbfa688bd6cb410e7fd56266e6cd7d1c7d6041e1a264: Status 404 returned error can't find the container with id 53ae8925b132dd02a1e2cbfa688bd6cb410e7fd56266e6cd7d1c7d6041e1a264
Sep 08 13:16:13.066748 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:13.066724    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f80b4d3_387c_4580_9899_0be849014b1e.slice/crio-c05000b9cd1381ab6298a46416d650ab215feb59fdeb75d0e358bb4be5b0cd09 WatchSource:0}: Error finding container c05000b9cd1381ab6298a46416d650ab215feb59fdeb75d0e358bb4be5b0cd09: Status 404 returned error can't find the container with id c05000b9cd1381ab6298a46416d650ab215feb59fdeb75d0e358bb4be5b0cd09
Sep 08 13:16:13.329051 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.328959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:13.329192 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:13.329099    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:13.329192 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:13.329163    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:14.32914865 +0000 UTC m=+4.032934481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:13.430130 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.430095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:13.430286 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:13.430263    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:13.430286 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:13.430283    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:13.430354 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:13.430292    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hzsss for pod openshift-network-diagnostics/network-check-target-f7wdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:13.430354 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:13.430344    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss podName:cd92145f-03d4-44bc-ab70-24feb2c337f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:14.43033037 +0000 UTC m=+4.134116197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzsss" (UniqueName: "kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss") pod "network-check-target-f7wdv" (UID: "cd92145f-03d4-44bc-ab70-24feb2c337f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:13.768370 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.768229    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:12 +0000 UTC" deadline="2028-02-05 22:43:36.205076975 +0000 UTC"
Sep 08 13:16:13.768370 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.768281    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12369h27m22.436798193s"
Sep 08 13:16:13.827302 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.827262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"e56960d4ec1624f3130633ba1a39a1c07e88e1f32196ffcc351b77722b4d8494"}
Sep 08 13:16:13.828826 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.828788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lnvw8" event={"ID":"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6","Type":"ContainerStarted","Data":"c349053a0780acd7b8bbeb1325836f2e00ef200279c5937a4e2ddb97477de159"}
Sep 08 13:16:13.831045 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.831020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-59gzp" event={"ID":"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2","Type":"ContainerStarted","Data":"a52067b5141ef8689dfb17892ad3131c6ebe94585d576b66408643e332713c4f"}
Sep 08 13:16:13.832895 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.832846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxllr" event={"ID":"2f80b4d3-387c-4580-9899-0be849014b1e","Type":"ContainerStarted","Data":"c05000b9cd1381ab6298a46416d650ab215feb59fdeb75d0e358bb4be5b0cd09"}
Sep 08 13:16:13.835103 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.835079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9" event={"ID":"d077d15f-0206-485e-a148-5d4c0a3e9ccf","Type":"ContainerStarted","Data":"53ae8925b132dd02a1e2cbfa688bd6cb410e7fd56266e6cd7d1c7d6041e1a264"}
Sep 08 13:16:13.838213 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.838189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnrbm" event={"ID":"c7e1ca70-26c8-46d3-ad76-7bea5f11a16c","Type":"ContainerStarted","Data":"673e5349de63319412c40e533214b05db56902d7405145e726488ed5cdb7bd14"}
Sep 08 13:16:13.839684 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.839647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" event={"ID":"93bed6de-bafb-4a04-aa8b-2a4f94ff1497","Type":"ContainerStarted","Data":"b8c707f49673a160301be2b8cd0b38d540aa0d4a5aaaa121312becf6a1363875"}
Sep 08 13:16:13.841548 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.841463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g9wzq" event={"ID":"b804ad43-3ef5-4847-914e-456974ad6d9b","Type":"ContainerStarted","Data":"a7c2d8ebb0ebced2268121b85ca3812160f72be6944e3836a4ef7311d8af9026"}
Sep 08 13:16:13.843915 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:13.843853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m7j8b" event={"ID":"72461732-cf32-4137-a68c-016f7d5d785a","Type":"ContainerStarted","Data":"733727f9020e01541715337d9296e4d9394123c2fa4b7b60004757325b0cd447"}
Sep 08 13:16:14.337881 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:14.337844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:14.338081 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:14.338057    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:14.338160 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:14.338122    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:16.338104271 +0000 UTC m=+6.041890101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:14.438463 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:14.438426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:14.438663 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:14.438579    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:14.438663 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:14.438598    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:14.438663 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:14.438611    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hzsss for pod openshift-network-diagnostics/network-check-target-f7wdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:14.438663 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:14.438683    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss podName:cd92145f-03d4-44bc-ab70-24feb2c337f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:16.438650596 +0000 UTC m=+6.142436441 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzsss" (UniqueName: "kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss") pod "network-check-target-f7wdv" (UID: "cd92145f-03d4-44bc-ab70-24feb2c337f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:14.769326 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:14.769248    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:12 +0000 UTC" deadline="2028-06-11 06:32:22.217067236 +0000 UTC"
Sep 08 13:16:14.769326 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:14.769290    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15401h16m7.447781831s"
Sep 08 13:16:14.820550 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:14.819757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:14.820550 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:14.819898    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:14.820550 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:14.820302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:14.820550 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:14.820397    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:16.354475 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:16.353893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:16.354475 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:16.354041    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:16.354475 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:16.354110    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:20.354091709 +0000 UTC m=+10.057877560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:16.455665 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:16.454927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:16.455665 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:16.455171    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:16.455665 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:16.455192    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:16.455665 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:16.455238    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hzsss for pod openshift-network-diagnostics/network-check-target-f7wdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:16.455665 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:16.455339    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss podName:cd92145f-03d4-44bc-ab70-24feb2c337f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:20.455316052 +0000 UTC m=+10.159101910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzsss" (UniqueName: "kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss") pod "network-check-target-f7wdv" (UID: "cd92145f-03d4-44bc-ab70-24feb2c337f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:16.831629 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:16.831547    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:16.831834 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:16.831803    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:16.832310 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:16.832035    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:16.833025 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:16.832944    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:18.818269 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:18.817764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:18.818269 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:18.817910    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:18.819016 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:18.818870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:18.819016 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:18.818980    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:20.385762 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:20.385725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:20.386155 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:20.385867    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:20.386155 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:20.385939    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:28.385918908 +0000 UTC m=+18.089704747 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:20.486423 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:20.486386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:20.486603 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:20.486573    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:20.486603 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:20.486596    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:20.486733 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:20.486609    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hzsss for pod openshift-network-diagnostics/network-check-target-f7wdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:20.486733 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:20.486681    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss podName:cd92145f-03d4-44bc-ab70-24feb2c337f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:28.486646931 +0000 UTC m=+18.190432763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzsss" (UniqueName: "kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss") pod "network-check-target-f7wdv" (UID: "cd92145f-03d4-44bc-ab70-24feb2c337f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:20.820738 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:20.820132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:20.820738 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:20.820233    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:20.820738 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:20.820542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:20.820738 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:20.820634    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:22.817728 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:22.817692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:22.818081 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:22.817703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:22.818081 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:22.817837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:22.818081 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:22.817895    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:24.816924 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:24.816892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:24.817307 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:24.817007    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:24.817404 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:24.817389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:24.817500 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:24.817476    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:26.817565 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:26.817536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:26.817953 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:26.817579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:26.817953 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:26.817695    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:26.817953 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:26.817755    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:27.866978 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:27.866945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal" event={"ID":"2e342b94596d41ae05df25539ced6fa9","Type":"ContainerStarted","Data":"960017d865e33231f1e5ab085b3c29be86c3d766dff22995c66d59e13b2580fe"}
Sep 08 13:16:28.441071 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:28.441035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:28.441239 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:28.441177    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:28.441286 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:28.441241    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:44.441222104 +0000 UTC m=+34.145007935 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:28.542100 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:28.542072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:28.542281 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:28.542251    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:28.542281 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:28.542274    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:28.542390 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:28.542285    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hzsss for pod openshift-network-diagnostics/network-check-target-f7wdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:28.542390 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:28.542338    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss podName:cd92145f-03d4-44bc-ab70-24feb2c337f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:44.54232004 +0000 UTC m=+34.246105868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzsss" (UniqueName: "kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss") pod "network-check-target-f7wdv" (UID: "cd92145f-03d4-44bc-ab70-24feb2c337f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:28.817573 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:28.817481    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:28.817573 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:28.817542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:28.817808 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:28.817739    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:28.817888 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:28.817850    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:30.818407 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:30.818373    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:30.818856 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:30.818505    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:30.818856 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:30.818559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:30.818856 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:30.818699    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:31.877348 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.876936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lnvw8" event={"ID":"d3fd4abc-ff7d-4512-a9f2-8103510ca5c6","Type":"ContainerStarted","Data":"89733db039820d140e03b2c70ddcf9782d18ef70d586ec00304c8245f175cb21"}
Sep 08 13:16:31.878865 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.878838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-59gzp" event={"ID":"fd8ad5d7-0e36-4ab4-b6e3-858afa8715b2","Type":"ContainerStarted","Data":"cfcb5d499424696cf164474e9d4639d0cea2401155b61292897d56ac01ffbbdf"}
Sep 08 13:16:31.880508 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.880482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal" event={"ID":"4084ecab986bde65b84b8c14d785fe7d","Type":"ContainerStarted","Data":"a148a62410e5d1f761a629460d97a6c2d775efc5d2d5938d9cf07cce2e000985"}
Sep 08 13:16:31.882510 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.882486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bxllr" event={"ID":"2f80b4d3-387c-4580-9899-0be849014b1e","Type":"ContainerStarted","Data":"3cd7c1846f547195efb45d373180fcd22d996efd5e10445646ff8f144e368d4d"}
Sep 08 13:16:31.884265 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.884239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9" event={"ID":"d077d15f-0206-485e-a148-5d4c0a3e9ccf","Type":"ContainerStarted","Data":"fff9d956c9b1dbe5c27840d1efcd9b39130f78af9c94d3354b5476c6f92a1780"}
Sep 08 13:16:31.885872 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.885847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnrbm" event={"ID":"c7e1ca70-26c8-46d3-ad76-7bea5f11a16c","Type":"ContainerStarted","Data":"6b1552a40018b9d6f72ecff55e932550037172b898a998fec2f55e53aebb99c6"}
Sep 08 13:16:31.887340 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.887318    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bed6de-bafb-4a04-aa8b-2a4f94ff1497" containerID="107aab32c78b4e10ef9e8e5dac51e039d5e0819955e5b67791ab2f3a7f7e6a56" exitCode=0
Sep 08 13:16:31.887430 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.887385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" event={"ID":"93bed6de-bafb-4a04-aa8b-2a4f94ff1497","Type":"ContainerDied","Data":"107aab32c78b4e10ef9e8e5dac51e039d5e0819955e5b67791ab2f3a7f7e6a56"}
Sep 08 13:16:31.889017 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.888986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g9wzq" event={"ID":"b804ad43-3ef5-4847-914e-456974ad6d9b","Type":"ContainerStarted","Data":"1d3343492e467121b15521362cd0d11125786e2186ea372108d0baf0eabb7742"}
Sep 08 13:16:31.890984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.890964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"3f139b1d4171650e669eddfd5834f01ead51a177f7d91cfa3783a6a67878279e"}
Sep 08 13:16:31.891070 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.891023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"859bf6ceff56e2da2f0422c698902aba83d1c8c64ecc9b5ab44ede964d3f09e6"}
Sep 08 13:16:31.894050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.894001    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lnvw8" podStartSLOduration=2.93345996 podStartE2EDuration="20.893984985s" totalImagesPullingTime="17.960525025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:31.89376968 +0000 UTC m=+21.597555531" watchObservedRunningTime="2026-09-08 13:16:31.893984985 +0000 UTC m=+21.597770838"
Sep 08 13:16:31.955074 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.955031    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wnrbm" podStartSLOduration=11.869121732 podStartE2EDuration="20.955018191s" totalImagesPullingTime="9.085896459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:31.954808353 +0000 UTC m=+21.658594203" watchObservedRunningTime="2026-09-08 13:16:31.955018191 +0000 UTC m=+21.658804040"
Sep 08 13:16:31.955442 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.955405    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-59gzp" podStartSLOduration=11.273968635 podStartE2EDuration="20.955398796s" totalImagesPullingTime="9.681430161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:31.93673725 +0000 UTC m=+21.640523101" watchObservedRunningTime="2026-09-08 13:16:31.955398796 +0000 UTC m=+21.659184645"
Sep 08 13:16:31.971194 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:31.971145    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bxllr" podStartSLOduration=3.038138379 podStartE2EDuration="20.971132869s" totalImagesPullingTime="17.93299449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:31.970887626 +0000 UTC m=+21.674673474" watchObservedRunningTime="2026-09-08 13:16:31.971132869 +0000 UTC m=+21.674918718"
Sep 08 13:16:32.024330 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.024275    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-211.ec2.internal" podStartSLOduration=20.024262018 podStartE2EDuration="20.024262018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:31.995769644 +0000 UTC m=+21.699555495" watchObservedRunningTime="2026-09-08 13:16:32.024262018 +0000 UTC m=+21.728047867"
Sep 08 13:16:32.804846 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:32.804559    2581 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:909d94acdf16bd26857849698608cb17db0c62ca3eab7d670ee858a6e839390e"
Sep 08 13:16:32.805077 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:32.804882    2581 kuberuntime_manager.go:1459] "Unhandled Error" err="container csi-node-driver-registrar start failed in pod aws-ebs-csi-driver-node-dbcm9_openshift-cluster-csi-drivers(d077d15f-0206-485e-a148-5d4c0a3e9ccf): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image" logger="UnhandledError"
Sep 08 13:16:32.816841 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.816816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:32.817005 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.816816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:32.817005 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:32.816940    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:32.817098 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:32.817050    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:32.895012 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.894961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m7j8b" event={"ID":"72461732-cf32-4137-a68c-016f7d5d785a","Type":"ContainerStarted","Data":"3142187a52c833c71c0adc6b01c85dd65550faf35e70b31353ad514a92a58265"}
Sep 08 13:16:32.898722 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.898697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"d70db99bd284ec71e0bd7b8fbfd852ca085125420cb25b1d892ef506756add2c"}
Sep 08 13:16:32.898811 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.898761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"4870b148b79c3aa000ab39fe28a8b4ff9376742a7039135226c770c087568a0b"}
Sep 08 13:16:32.898811 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.898796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"3344363f6b5c38770fbd6f08d3ab63662d43c920ec98125f70ce1f40e138cdfa"}
Sep 08 13:16:32.898892 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.898828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"4030b501d61b6847bee75967f806df42884d3558ef99eeba87ccb0c284776344"}
Sep 08 13:16:32.900584 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.900559    2581 generic.go:356] "Generic (PLEG): container finished" podID="2e342b94596d41ae05df25539ced6fa9" containerID="66913d0d3967b48154252cc12fb0c4b8051e535ca6298582d57e7ca7c588f5f8" exitCode=0
Sep 08 13:16:32.900699 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.900653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal" event={"ID":"2e342b94596d41ae05df25539ced6fa9","Type":"ContainerDied","Data":"66913d0d3967b48154252cc12fb0c4b8051e535ca6298582d57e7ca7c588f5f8"}
Sep 08 13:16:32.912489 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.912450    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m7j8b" podStartSLOduration=13.791025296 podStartE2EDuration="22.912438525s" totalImagesPullingTime="9.121413229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:32.912242311 +0000 UTC m=+22.616028161" watchObservedRunningTime="2026-09-08 13:16:32.912438525 +0000 UTC m=+22.616224373"
Sep 08 13:16:32.912656 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:32.912627    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g9wzq" podStartSLOduration=3.959646286 podStartE2EDuration="21.9126187s" totalImagesPullingTime="17.952972414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:32.025083136 +0000 UTC m=+21.728868985" watchObservedRunningTime="2026-09-08 13:16:32.9126187 +0000 UTC m=+22.616404551"
Sep 08 13:16:33.904897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:33.904861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal" event={"ID":"2e342b94596d41ae05df25539ced6fa9","Type":"ContainerStarted","Data":"008377101abbfb513cf3d679c9ef26157718eca7adb900e70276a289a21eb87a"}
Sep 08 13:16:33.922805 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:33.922760    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-211.ec2.internal" podStartSLOduration=21.922745805 podStartE2EDuration="21.922745805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:33.922699891 +0000 UTC m=+23.626485741" watchObservedRunningTime="2026-09-08 13:16:33.922745805 +0000 UTC m=+23.626531655"
Sep 08 13:16:34.376631 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:34.376564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:34.525606 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:34.525581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:34.526274 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:34.526252    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:34.817607 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:34.817531    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:34.817777 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:34.817531    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:34.817777 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:34.817652    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:34.817887 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:34.817794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:34.910818 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:34.910780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"4e407bbb23d503b50dcdba55659503631bb3d4fdf462afb6f413600816dbffe2"}
Sep 08 13:16:34.911500 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:34.911478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wnrbm"
Sep 08 13:16:36.816875 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.816850    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:36.817201 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:36.816964    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:36.817201 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.817020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:36.817201 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:36.817105    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:36.918433 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.918179    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bed6de-bafb-4a04-aa8b-2a4f94ff1497" containerID="e2df9e1497ad331cd4752cac10926088b6714b616bea34e5d1c52edfc8987e77" exitCode=0
Sep 08 13:16:36.918555 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.918265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" event={"ID":"93bed6de-bafb-4a04-aa8b-2a4f94ff1497","Type":"ContainerDied","Data":"e2df9e1497ad331cd4752cac10926088b6714b616bea34e5d1c52edfc8987e77"}
Sep 08 13:16:36.923732 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.923704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" event={"ID":"0c2245f7-7abe-45bd-8ab1-d500057643ff","Type":"ContainerStarted","Data":"8393ae6892be595da803556aae1d477c0682f1f934c87c1749a323579ab1a6eb"}
Sep 08 13:16:36.924182 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.923996    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:36.924182 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.924067    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:36.924182 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.924120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:36.941979 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.941933    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:36.942330 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.942316    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:16:36.970912 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:36.970873    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x" podStartSLOduration=7.147857297 podStartE2EDuration="25.970859684s" totalImagesPullingTime="18.823002387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:36.969655961 +0000 UTC m=+26.673441810" watchObservedRunningTime="2026-09-08 13:16:36.970859684 +0000 UTC m=+26.674645533"
Sep 08 13:16:37.035125 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:37.035096    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9" podUID="d077d15f-0206-485e-a148-5d4c0a3e9ccf"
Sep 08 13:16:37.927422 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:37.927381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9" event={"ID":"d077d15f-0206-485e-a148-5d4c0a3e9ccf","Type":"ContainerStarted","Data":"8e505c7b1afa714920ea75fc3cf824b4abc8af4ce9aca6c751797dce654fb43c"}
Sep 08 13:16:37.928570 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:37.928517    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:909d94acdf16bd26857849698608cb17db0c62ca3eab7d670ee858a6e839390e\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9" podUID="d077d15f-0206-485e-a148-5d4c0a3e9ccf"
Sep 08 13:16:38.643501 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:38.643465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f7wdv"]
Sep 08 13:16:38.643756 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:38.643605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:38.643756 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:38.643730    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:38.646948 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:38.646918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mpqn2"]
Sep 08 13:16:38.647101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:38.647044    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:38.647262 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:38.647233    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:38.930834 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:38.930779    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bed6de-bafb-4a04-aa8b-2a4f94ff1497" containerID="f1769362afd4986b8ee37e717359a5339b008f407fcee520d685f78b9ad18cb5" exitCode=0
Sep 08 13:16:38.931443 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:38.930869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" event={"ID":"93bed6de-bafb-4a04-aa8b-2a4f94ff1497","Type":"ContainerDied","Data":"f1769362afd4986b8ee37e717359a5339b008f407fcee520d685f78b9ad18cb5"}
Sep 08 13:16:38.932423 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:38.932388    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:909d94acdf16bd26857849698608cb17db0c62ca3eab7d670ee858a6e839390e\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 502 Bad Gateway; artifact err: provided artifact is a container image\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9" podUID="d077d15f-0206-485e-a148-5d4c0a3e9ccf"
Sep 08 13:16:40.818081 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:40.818012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:40.818384 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:40.818094    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:40.818384 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:40.818208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:40.818384 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:40.818355    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:42.817775 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:42.817587    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:42.818091 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:42.817634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:42.818091 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:42.817862    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:42.818091 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:42.817931    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:42.941750 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:42.941723    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bed6de-bafb-4a04-aa8b-2a4f94ff1497" containerID="720b1ea9b9c567047dd58aa44bf39d3f3066c9ced8acd43869a9ee50746a7d46" exitCode=0
Sep 08 13:16:42.941863 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:42.941781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" event={"ID":"93bed6de-bafb-4a04-aa8b-2a4f94ff1497","Type":"ContainerDied","Data":"720b1ea9b9c567047dd58aa44bf39d3f3066c9ced8acd43869a9ee50746a7d46"}
Sep 08 13:16:44.453407 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:44.453373    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:44.453881 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:44.453495    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:44.453881 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:44.453555    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:16.453538687 +0000 UTC m=+66.157324531 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:44.554496 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:44.554468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:44.554635 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:44.554617    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:44.554697 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:44.554641    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:44.554697 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:44.554650    2581 projected.go:196] Error preparing data for projected volume kube-api-access-hzsss for pod openshift-network-diagnostics/network-check-target-f7wdv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:44.554793 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:44.554719    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss podName:cd92145f-03d4-44bc-ab70-24feb2c337f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:16.554705425 +0000 UTC m=+66.258491253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzsss" (UniqueName: "kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss") pod "network-check-target-f7wdv" (UID: "cd92145f-03d4-44bc-ab70-24feb2c337f4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:44.817063 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:44.817037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:44.817198 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:44.817037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:44.817198 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:44.817147    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:16:44.817270 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:44.817206    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f7wdv" podUID="cd92145f-03d4-44bc-ab70-24feb2c337f4"
Sep 08 13:16:45.079944 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.079877    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-211.ec2.internal" event="NodeReady"
Sep 08 13:16:45.080077 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.079975    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:16:45.114904 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.114878    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-687dc7f5fd-c4fhz"]
Sep 08 13:16:45.138541 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.138520    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ntxdb"]
Sep 08 13:16:45.138647 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.138628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.140758 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.140742    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 13:16:45.140843 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.140805    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p6qj7\""
Sep 08 13:16:45.141087 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.141073    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 13:16:45.141239 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.141222    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 13:16:45.162716 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.162694    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-687dc7f5fd-c4fhz"]
Sep 08 13:16:45.162815 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.162734    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qx8ch"]
Sep 08 13:16:45.162815 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.162761    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.163369 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.163352    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 13:16:45.165649 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.165632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:16:45.165649 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.165640    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8sm9m\""
Sep 08 13:16:45.165884 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.165870    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:16:45.179062 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.179044    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ntxdb"]
Sep 08 13:16:45.179163 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.179079    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qx8ch"]
Sep 08 13:16:45.179163 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.179101    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-lfltj"]
Sep 08 13:16:45.179163 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.179147    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qx8ch"
Sep 08 13:16:45.184113 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.184094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:16:45.184722 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.184704    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:16:45.184972 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.184957    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z887n\""
Sep 08 13:16:45.185012 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.184960    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:16:45.194905 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.194886    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lfltj"]
Sep 08 13:16:45.194983 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.194976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.197754 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.197738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:16:45.259246 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259219    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.259246 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-registry-certificates\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.259459 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.259459 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-image-registry-private-configuration\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.259459 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9nn\" (UniqueName: \"kubernetes.io/projected/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-kube-api-access-2m9nn\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.259459 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86c78164-13cb-4515-a395-08a81ccc9594-ca-trust-extracted\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.259459 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-installation-pull-secrets\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.259459 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq8d9\" (UniqueName: \"kubernetes.io/projected/a1e7cc14-c9c0-4576-931e-53bc31b9326b-kube-api-access-hq8d9\") pod \"ingress-canary-qx8ch\" (UID: \"a1e7cc14-c9c0-4576-931e-53bc31b9326b\") " pod="openshift-ingress-canary/ingress-canary-qx8ch"
Sep 08 13:16:45.259661 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259483    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5w7m\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-kube-api-access-h5w7m\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.259661 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259500    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-tmp-dir\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.259661 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-bound-sa-token\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.259661 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-trusted-ca\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.259661 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-config-volume\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.259661 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.259588    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e7cc14-c9c0-4576-931e-53bc31b9326b-cert\") pod \"ingress-canary-qx8ch\" (UID: \"a1e7cc14-c9c0-4576-931e-53bc31b9326b\") " pod="openshift-ingress-canary/ingress-canary-qx8ch"
Sep 08 13:16:45.360553 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq8d9\" (UniqueName: \"kubernetes.io/projected/a1e7cc14-c9c0-4576-931e-53bc31b9326b-kube-api-access-hq8d9\") pod \"ingress-canary-qx8ch\" (UID: \"a1e7cc14-c9c0-4576-931e-53bc31b9326b\") " pod="openshift-ingress-canary/ingress-canary-qx8ch"
Sep 08 13:16:45.360553 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360519    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02436e3b-bde9-4934-98a4-91de9fec7159-original-pull-secret\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.360760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360566    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5w7m\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-kube-api-access-h5w7m\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.360760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-tmp-dir\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.360760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02436e3b-bde9-4934-98a4-91de9fec7159-kubelet-config\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.360760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-bound-sa-token\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.360760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360693    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-trusted-ca\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.360760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-config-volume\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.360760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e7cc14-c9c0-4576-931e-53bc31b9326b-cert\") pod \"ingress-canary-qx8ch\" (UID: \"a1e7cc14-c9c0-4576-931e-53bc31b9326b\") " pod="openshift-ingress-canary/ingress-canary-qx8ch"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-registry-certificates\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-image-registry-private-configuration\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9nn\" (UniqueName: \"kubernetes.io/projected/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-kube-api-access-2m9nn\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86c78164-13cb-4515-a395-08a81ccc9594-ca-trust-extracted\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.360895    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.360914    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-installation-pull-secrets\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02436e3b-bde9-4934-98a4-91de9fec7159-dbus\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.360954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-tmp-dir\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.361191 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.360971    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:45.860949479 +0000 UTC m=+35.564735308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:16:45.361744 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.361391    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86c78164-13cb-4515-a395-08a81ccc9594-ca-trust-extracted\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.361744 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.361440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-config-volume\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.361744 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.361485    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:45.361744 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.361547    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:45.861532866 +0000 UTC m=+35.565318693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:45.361744 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.361648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-registry-certificates\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.361943 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.361755    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-trusted-ca\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.365059 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.365031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-installation-pull-secrets\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.365157 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.365076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-image-registry-private-configuration\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.365198 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.365167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1e7cc14-c9c0-4576-931e-53bc31b9326b-cert\") pod \"ingress-canary-qx8ch\" (UID: \"a1e7cc14-c9c0-4576-931e-53bc31b9326b\") " pod="openshift-ingress-canary/ingress-canary-qx8ch"
Sep 08 13:16:45.373524 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.373488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9nn\" (UniqueName: \"kubernetes.io/projected/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-kube-api-access-2m9nn\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.373822 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.373805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-bound-sa-token\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.375315 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.375296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq8d9\" (UniqueName: \"kubernetes.io/projected/a1e7cc14-c9c0-4576-931e-53bc31b9326b-kube-api-access-hq8d9\") pod \"ingress-canary-qx8ch\" (UID: \"a1e7cc14-c9c0-4576-931e-53bc31b9326b\") " pod="openshift-ingress-canary/ingress-canary-qx8ch"
Sep 08 13:16:45.375414 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.375397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5w7m\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-kube-api-access-h5w7m\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.461602 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.461578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02436e3b-bde9-4934-98a4-91de9fec7159-kubelet-config\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.461939 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.461632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02436e3b-bde9-4934-98a4-91de9fec7159-dbus\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.461939 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.461656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02436e3b-bde9-4934-98a4-91de9fec7159-original-pull-secret\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.461939 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.461729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/02436e3b-bde9-4934-98a4-91de9fec7159-kubelet-config\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.461939 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.461793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/02436e3b-bde9-4934-98a4-91de9fec7159-dbus\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.463544 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.463528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/02436e3b-bde9-4934-98a4-91de9fec7159-original-pull-secret\") pod \"global-pull-secret-syncer-lfltj\" (UID: \"02436e3b-bde9-4934-98a4-91de9fec7159\") " pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.488596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.488578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qx8ch"
Sep 08 13:16:45.502894 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.502878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lfltj"
Sep 08 13:16:45.812686 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.812491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lfltj"]
Sep 08 13:16:45.815951 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.815926    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qx8ch"]
Sep 08 13:16:45.816544 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:45.816514    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02436e3b_bde9_4934_98a4_91de9fec7159.slice/crio-c9a43bf906df67717014d1810f536bb3a94afe4ff576a5319fba0ad783fae5b8 WatchSource:0}: Error finding container c9a43bf906df67717014d1810f536bb3a94afe4ff576a5319fba0ad783fae5b8: Status 404 returned error can't find the container with id c9a43bf906df67717014d1810f536bb3a94afe4ff576a5319fba0ad783fae5b8
Sep 08 13:16:45.820592 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:45.820567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e7cc14_c9c0_4576_931e_53bc31b9326b.slice/crio-7e4c8552ad63b51d7dc6a28f59960c872665a886673d41f567f1540c55170f1a WatchSource:0}: Error finding container 7e4c8552ad63b51d7dc6a28f59960c872665a886673d41f567f1540c55170f1a: Status 404 returned error can't find the container with id 7e4c8552ad63b51d7dc6a28f59960c872665a886673d41f567f1540c55170f1a
Sep 08 13:16:45.865258 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.865234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:45.865364 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.865272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:45.865423 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.865400    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:45.865423 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.865406    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:45.865423 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.865423    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:16:45.865572 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.865449    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:46.865431637 +0000 UTC m=+36.569217483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:45.865572 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:45.865486    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:46.865468285 +0000 UTC m=+36.569254116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:16:45.949334 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.949303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lfltj" event={"ID":"02436e3b-bde9-4934-98a4-91de9fec7159","Type":"ContainerStarted","Data":"c9a43bf906df67717014d1810f536bb3a94afe4ff576a5319fba0ad783fae5b8"}
Sep 08 13:16:45.950374 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:45.950353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qx8ch" event={"ID":"a1e7cc14-c9c0-4576-931e-53bc31b9326b","Type":"ContainerStarted","Data":"7e4c8552ad63b51d7dc6a28f59960c872665a886673d41f567f1540c55170f1a"}
Sep 08 13:16:46.817859 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.817827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:16:46.818311 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.817838    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:16:46.820880 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.820855    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.820880 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.820877    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:16:46.821629 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.821612    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2p8w4\""
Sep 08 13:16:46.822444 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.821854    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.822444 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.822024    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-486d2\""
Sep 08 13:16:46.872610 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.872579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:46.872763 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.872620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:46.872763 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:46.872747    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:46.872763 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:46.872756    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:46.872915 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:46.872773    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:16:46.872915 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:46.872791    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.872779399 +0000 UTC m=+38.576565227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:46.872915 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:46.872825    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.872807899 +0000 UTC m=+38.576593729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:16:46.879857 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.879828    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"]
Sep 08 13:16:46.888589 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.888569    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"]
Sep 08 13:16:46.889454 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.888723    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"
Sep 08 13:16:46.892099 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.892079    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9qrzd\""
Sep 08 13:16:46.892198 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.892099    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.892259 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.892246    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.892696 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.892612    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 08 13:16:46.892981 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.892959    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 13:16:46.901897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.901478    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"]
Sep 08 13:16:46.901897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.901523    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"]
Sep 08 13:16:46.901897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.901612    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:46.901897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.901814    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"]
Sep 08 13:16:46.904370 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.904352    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 13:16:46.910227 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.910208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:46.923076 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.923053    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 13:16:46.923371 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.923353    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 13:16:46.923887 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.923849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 13:16:46.924720 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.924699    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 13:16:46.930350 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.930328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"]
Sep 08 13:16:46.973938 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.973904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc83150-c87b-4013-b6f4-f23f331d62c9-tmp\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:46.974104 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.973965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhrp\" (UniqueName: \"kubernetes.io/projected/c3c969ab-d328-4c83-99ce-8864535b2bc2-kube-api-access-whhrp\") pod \"managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp\" (UID: \"c3c969ab-d328-4c83-99ce-8864535b2bc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"
Sep 08 13:16:46.974171 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.974091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4k5\" (UniqueName: \"kubernetes.io/projected/1fc83150-c87b-4013-b6f4-f23f331d62c9-kube-api-access-4n4k5\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:46.974171 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.974136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1fc83150-c87b-4013-b6f4-f23f331d62c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:46.974275 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:46.974217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3c969ab-d328-4c83-99ce-8864535b2bc2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp\" (UID: \"c3c969ab-d328-4c83-99ce-8864535b2bc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"
Sep 08 13:16:47.074935 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.074857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.075078 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.074937    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhrp\" (UniqueName: \"kubernetes.io/projected/c3c969ab-d328-4c83-99ce-8864535b2bc2-kube-api-access-whhrp\") pod \"managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp\" (UID: \"c3c969ab-d328-4c83-99ce-8864535b2bc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"
Sep 08 13:16:47.075078 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.074965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.075192 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075055    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4k5\" (UniqueName: \"kubernetes.io/projected/1fc83150-c87b-4013-b6f4-f23f331d62c9-kube-api-access-4n4k5\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:47.075192 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-ca\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.075192 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1fc83150-c87b-4013-b6f4-f23f331d62c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:47.075192 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-hub\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.075192 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.075403 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3c969ab-d328-4c83-99ce-8864535b2bc2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp\" (UID: \"c3c969ab-d328-4c83-99ce-8864535b2bc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"
Sep 08 13:16:47.075403 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjjj\" (UniqueName: \"kubernetes.io/projected/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-kube-api-access-jsjjj\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.075403 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc83150-c87b-4013-b6f4-f23f331d62c9-tmp\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:47.075591 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.075567    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1fc83150-c87b-4013-b6f4-f23f331d62c9-tmp\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:47.079133 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.079087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3c969ab-d328-4c83-99ce-8864535b2bc2-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp\" (UID: \"c3c969ab-d328-4c83-99ce-8864535b2bc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"
Sep 08 13:16:47.079241 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.079167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/1fc83150-c87b-4013-b6f4-f23f331d62c9-klusterlet-config\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:47.094381 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.094350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4k5\" (UniqueName: \"kubernetes.io/projected/1fc83150-c87b-4013-b6f4-f23f331d62c9-kube-api-access-4n4k5\") pod \"klusterlet-addon-workmgr-5794954d67-l9czd\" (UID: \"1fc83150-c87b-4013-b6f4-f23f331d62c9\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:47.095717 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.095659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhrp\" (UniqueName: \"kubernetes.io/projected/c3c969ab-d328-4c83-99ce-8864535b2bc2-kube-api-access-whhrp\") pod \"managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp\" (UID: \"c3c969ab-d328-4c83-99ce-8864535b2bc2\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"
Sep 08 13:16:47.176178 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.176143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-ca\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.176311 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.176196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-hub\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.176311 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.176223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.176311 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.176279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjjj\" (UniqueName: \"kubernetes.io/projected/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-kube-api-access-jsjjj\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.176406 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.176314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.176406 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.176359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.177522 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.177464    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-additional-service-ca\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.179919 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.179896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.180026 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.179988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.181168 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.181141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-ca\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.182284 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.182259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-hub\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.187624 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.187595    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjjj\" (UniqueName: \"kubernetes.io/projected/f31a1fe3-0ae3-41d3-81a0-926e5a2f7510-kube-api-access-jsjjj\") pod \"cluster-proxy-proxy-agent-69d685bffc-znwsg\" (UID: \"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.207430 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.207410    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"
Sep 08 13:16:47.217576 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.217546    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:16:47.226877 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.226856    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:16:47.528956 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.528926    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp"]
Sep 08 13:16:47.541879 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.541853    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"]
Sep 08 13:16:47.546264 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:47.546239    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc83150_c87b_4013_b6f4_f23f331d62c9.slice/crio-1f24f6c2a2d918b3238fe3d00b5cd6881ca6a7406f57bbee0d5d21c58851c65b WatchSource:0}: Error finding container 1f24f6c2a2d918b3238fe3d00b5cd6881ca6a7406f57bbee0d5d21c58851c65b: Status 404 returned error can't find the container with id 1f24f6c2a2d918b3238fe3d00b5cd6881ca6a7406f57bbee0d5d21c58851c65b
Sep 08 13:16:47.564968 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.564926    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"]
Sep 08 13:16:47.568548 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:16:47.568514    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf31a1fe3_0ae3_41d3_81a0_926e5a2f7510.slice/crio-e686514e6d7a7239d244a37d7a15d24a8af9effbc4f438410c7acfe89d706f79 WatchSource:0}: Error finding container e686514e6d7a7239d244a37d7a15d24a8af9effbc4f438410c7acfe89d706f79: Status 404 returned error can't find the container with id e686514e6d7a7239d244a37d7a15d24a8af9effbc4f438410c7acfe89d706f79
Sep 08 13:16:47.957704 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.957609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd" event={"ID":"1fc83150-c87b-4013-b6f4-f23f331d62c9","Type":"ContainerStarted","Data":"1f24f6c2a2d918b3238fe3d00b5cd6881ca6a7406f57bbee0d5d21c58851c65b"}
Sep 08 13:16:47.959782 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.959756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp" event={"ID":"c3c969ab-d328-4c83-99ce-8864535b2bc2","Type":"ContainerStarted","Data":"0060e66231d56e5844efe9ee939e57f01784b6919a19525d69046f284e480b1a"}
Sep 08 13:16:47.961041 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:47.961018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" event={"ID":"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510","Type":"ContainerStarted","Data":"e686514e6d7a7239d244a37d7a15d24a8af9effbc4f438410c7acfe89d706f79"}
Sep 08 13:16:48.894096 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:48.894004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:48.894096 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:48.894057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:48.894318 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:48.894192    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:48.894318 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:48.894211    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:48.894318 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:48.894214    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:16:48.894318 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:48.894272    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:52.894253113 +0000 UTC m=+42.598038955 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:48.894318 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:48.894288    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:52.894282116 +0000 UTC m=+42.598067942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:16:52.933893 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:52.933854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:16:52.933893 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:16:52.933900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:16:52.934427 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:52.934080    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:52.934427 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:52.934130    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:00.934116064 +0000 UTC m=+50.637901890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:52.934538 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:52.934506    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:52.934538 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:52.934519    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:16:52.934636 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:16:52.934552    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:00.934542047 +0000 UTC m=+50.638327875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:17:00.135714 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:00.135422    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:17:00.808261 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:00.808182    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:17:00.135449183Z","UUID":"d0f8e907-a6e8-48f2-989f-c0c50cce4356","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:17:00.809530 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:00.809510    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:17:00.809642 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:00.809537    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:17:00.995430 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:00.995394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:17:00.995430 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:00.995431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:17:00.995642 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:00.995535    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:00.995642 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:00.995546    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:17:00.995642 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:00.995566    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:17:00.995642 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:00.995584    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:16.995570666 +0000 UTC m=+66.699356493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:00.995642 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:00.995618    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:16.99560083 +0000 UTC m=+66.699386676 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:17:00.999730 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:00.999700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9" event={"ID":"d077d15f-0206-485e-a148-5d4c0a3e9ccf","Type":"ContainerStarted","Data":"674360acf8e65045d9413deb70a4701f189a4449f0e5642ffc43b91ba34eea4a"}
Sep 08 13:17:01.001142 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.001116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qx8ch" event={"ID":"a1e7cc14-c9c0-4576-931e-53bc31b9326b","Type":"ContainerStarted","Data":"6048976458e288e3783b365424c42418a2401c926383dccc6e7c6ea7a66b13c3"}
Sep 08 13:17:01.003768 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.003743    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bed6de-bafb-4a04-aa8b-2a4f94ff1497" containerID="b4975265313e8be975cf602fac818c581318d42e4fd6c60a5d561510db523454" exitCode=0
Sep 08 13:17:01.003866 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.003823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" event={"ID":"93bed6de-bafb-4a04-aa8b-2a4f94ff1497","Type":"ContainerDied","Data":"b4975265313e8be975cf602fac818c581318d42e4fd6c60a5d561510db523454"}
Sep 08 13:17:01.005270 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.005243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd" event={"ID":"1fc83150-c87b-4013-b6f4-f23f331d62c9","Type":"ContainerStarted","Data":"72937019c32d6cbe160b397f4688e8bb8df26897d7ed0afb72c791bc38b85850"}
Sep 08 13:17:01.005436 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.005413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:17:01.006871 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.006846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp" event={"ID":"c3c969ab-d328-4c83-99ce-8864535b2bc2","Type":"ContainerStarted","Data":"a4afcb9a1901bccd7469565e11b284a4cef325ae060b1e36da147f944d43b742"}
Sep 08 13:17:01.007373 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.007356    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:17:01.008407 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.008389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lfltj" event={"ID":"02436e3b-bde9-4934-98a4-91de9fec7159","Type":"ContainerStarted","Data":"e3d1a84d7aa0c222d363ef22d91b58a9636a6a17161f12c3ce472c73c71447fe"}
Sep 08 13:17:01.012530 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.012509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" event={"ID":"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510","Type":"ContainerStarted","Data":"b81ca696fb5bf40fbb4a764a48c0fcca5a91402a9e193fe065a971d64308b74f"}
Sep 08 13:17:01.012604 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.012543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" event={"ID":"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510","Type":"ContainerStarted","Data":"6a60cc381a05fa722a18f96229bcfd90f77d46d11ab95a6db29dd0c2c59ce35a"}
Sep 08 13:17:01.012604 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.012561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" event={"ID":"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510","Type":"ContainerStarted","Data":"fa920d0f641a553301cfedcafc8e421202e2c6ba3762e0eb68c8146be858a3e9"}
Sep 08 13:17:01.020702 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.020649    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dbcm9" podStartSLOduration=12.518617415 podStartE2EDuration="50.020637745s" totalImagesPullingTime="37.50202033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-08 13:17:01.020362666 +0000 UTC m=+50.724148516" watchObservedRunningTime="2026-09-08 13:17:01.020637745 +0000 UTC m=+50.724423594"
Sep 08 13:17:01.037759 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.037713    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qx8ch" podStartSLOduration=1.992481347 podStartE2EDuration="16.03769681s" totalImagesPullingTime="14.045215463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:01.036767806 +0000 UTC m=+50.740553647" watchObservedRunningTime="2026-09-08 13:17:01.03769681 +0000 UTC m=+50.741482672"
Sep 08 13:17:01.054379 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.054343    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd" podStartSLOduration=2.733823763 podStartE2EDuration="15.054331143s" totalImagesPullingTime="12.32050738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:01.053302346 +0000 UTC m=+50.757088195" watchObservedRunningTime="2026-09-08 13:17:01.054331143 +0000 UTC m=+50.758116986"
Sep 08 13:17:01.086328 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.086280    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lfltj" podStartSLOduration=2.588279131 podStartE2EDuration="16.086265892s" totalImagesPullingTime="13.497986761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:01.069805032 +0000 UTC m=+50.773590882" watchObservedRunningTime="2026-09-08 13:17:01.086265892 +0000 UTC m=+50.790051740"
Sep 08 13:17:01.086430 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.086357    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp" podStartSLOduration=2.753765124 podStartE2EDuration="15.086352303s" totalImagesPullingTime="12.332587179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:01.085781745 +0000 UTC m=+50.789567596" watchObservedRunningTime="2026-09-08 13:17:01.086352303 +0000 UTC m=+50.790138150"
Sep 08 13:17:01.127265 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:01.127221    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" podStartSLOduration=2.82858173 podStartE2EDuration="15.127207729s" totalImagesPullingTime="12.298625999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:01.126578825 +0000 UTC m=+50.830364673" watchObservedRunningTime="2026-09-08 13:17:01.127207729 +0000 UTC m=+50.830993576"
Sep 08 13:17:02.018424 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:02.018381    2581 generic.go:356] "Generic (PLEG): container finished" podID="93bed6de-bafb-4a04-aa8b-2a4f94ff1497" containerID="0f8f403aaec6692721e9d4085ce028514add55068c3fb20d1ad3e17bdf7f69d7" exitCode=0
Sep 08 13:17:02.018871 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:02.018468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" event={"ID":"93bed6de-bafb-4a04-aa8b-2a4f94ff1497","Type":"ContainerDied","Data":"0f8f403aaec6692721e9d4085ce028514add55068c3fb20d1ad3e17bdf7f69d7"}
Sep 08 13:17:03.024994 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:03.024949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" event={"ID":"93bed6de-bafb-4a04-aa8b-2a4f94ff1497","Type":"ContainerStarted","Data":"1a43f4da96fa4a650a6f328d585d04524a3f447d05435761aa315cf421e96dc8"}
Sep 08 13:17:03.053313 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:03.053271    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sgl7l" podStartSLOduration=16.646303525 podStartE2EDuration="52.053256448s" totalImagesPullingTime="35.406952923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:03.052279601 +0000 UTC m=+52.756065444" watchObservedRunningTime="2026-09-08 13:17:03.053256448 +0000 UTC m=+52.757042297"
Sep 08 13:17:08.941864 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:08.941832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c2n4x"
Sep 08 13:17:16.509738 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:16.509696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:17:16.512163 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:16.512144    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:17:16.519959 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:16.519936    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:17:16.520047 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:16.520002    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:20.519984209 +0000 UTC m=+130.223770036 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : secret "metrics-daemon-secret" not found
Sep 08 13:17:16.610430 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:16.610384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:17:16.613025 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:16.613006    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:17:16.623158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:16.623132    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:17:16.633634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:16.633609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsss\" (UniqueName: \"kubernetes.io/projected/cd92145f-03d4-44bc-ab70-24feb2c337f4-kube-api-access-hzsss\") pod \"network-check-target-f7wdv\" (UID: \"cd92145f-03d4-44bc-ab70-24feb2c337f4\") " pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:17:16.834014 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:16.833988    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-486d2\""
Sep 08 13:17:16.842458 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:16.842429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:17:17.014307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:17.014265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:17:17.014307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:17.014308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:17:17.014511 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:17.014410    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:17.014511 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:17.014414    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:17:17.014511 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:17.014432    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:17:17.014511 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:17.014476    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:49.01445679 +0000 UTC m=+98.718242629 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:17:17.014511 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:17.014492    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:49.014484662 +0000 UTC m=+98.718270493 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:17.118398 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:17.118325    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f7wdv"]
Sep 08 13:17:17.121713 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:17:17.121683    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd92145f_03d4_44bc_ab70_24feb2c337f4.slice/crio-7584afba716e0813e1b85f65f66131403c49cced439568fe42a758f886f6c351 WatchSource:0}: Error finding container 7584afba716e0813e1b85f65f66131403c49cced439568fe42a758f886f6c351: Status 404 returned error can't find the container with id 7584afba716e0813e1b85f65f66131403c49cced439568fe42a758f886f6c351
Sep 08 13:17:18.072688 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:18.072638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f7wdv" event={"ID":"cd92145f-03d4-44bc-ab70-24feb2c337f4","Type":"ContainerStarted","Data":"7584afba716e0813e1b85f65f66131403c49cced439568fe42a758f886f6c351"}
Sep 08 13:17:20.079716 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:20.079687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f7wdv" event={"ID":"cd92145f-03d4-44bc-ab70-24feb2c337f4","Type":"ContainerStarted","Data":"e01641ec6eebbd68a3dba5bd899fae6cb7c5d939940843e138b10ccbec9e4454"}
Sep 08 13:17:20.080092 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:20.079816    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:17:20.100890 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:20.100847    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f7wdv" podStartSLOduration=66.247441534 podStartE2EDuration="1m9.100832408s" totalImagesPullingTime="2.853390874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:20.099060789 +0000 UTC m=+69.802846682" watchObservedRunningTime="2026-09-08 13:17:20.100832408 +0000 UTC m=+69.804618257"
Sep 08 13:17:49.044306 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:49.044276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:17:49.044823 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:49.044314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:17:49.044823 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:49.044424    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:17:49.044823 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:49.044441    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:17:49.044823 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:49.044452    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:49.044823 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:49.044492    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:53.044477982 +0000 UTC m=+162.748263809 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:17:49.044823 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:17:49.044505    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:53.044500194 +0000 UTC m=+162.748286021 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:51.085695 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:17:51.085645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f7wdv"
Sep 08 13:18:20.564402 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:18:20.564362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:18:20.564889 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:20.564500    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:18:20.564889 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:20.564570    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs podName:1a5ec803-ab0d-4166-bf48-77a72a26b284 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:22.564556021 +0000 UTC m=+252.268341848 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs") pod "network-metrics-daemon-mpqn2" (UID: "1a5ec803-ab0d-4166-bf48-77a72a26b284") : secret "metrics-daemon-secret" not found
Sep 08 13:18:48.148145 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:48.148090    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz" podUID="86c78164-13cb-4515-a395-08a81ccc9594"
Sep 08 13:18:48.171363 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:48.171334    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ntxdb" podUID="c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810"
Sep 08 13:18:48.330306 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:18:48.330276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:18:48.330424 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:18:48.330282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:18:49.838245 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:49.838194    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mpqn2" podUID="1a5ec803-ab0d-4166-bf48-77a72a26b284"
Sep 08 13:18:53.083453 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:18:53.083406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") pod \"image-registry-687dc7f5fd-c4fhz\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") " pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:18:53.083453 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:18:53.083457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:18:53.083982 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:53.083559    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:53.083982 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:53.083578    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-687dc7f5fd-c4fhz: secret "image-registry-tls" not found
Sep 08 13:18:53.083982 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:53.083598    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:18:53.083982 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:53.083702    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls podName:c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:55.083662619 +0000 UTC m=+284.787448449 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls") pod "dns-default-ntxdb" (UID: "c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810") : secret "dns-default-metrics-tls" not found
Sep 08 13:18:53.083982 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:18:53.083723    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls podName:86c78164-13cb-4515-a395-08a81ccc9594 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:55.083714255 +0000 UTC m=+284.787500082 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls") pod "image-registry-687dc7f5fd-c4fhz" (UID: "86c78164-13cb-4515-a395-08a81ccc9594") : secret "image-registry-tls" not found
Sep 08 13:19:00.368286 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:00.368250    2581 generic.go:356] "Generic (PLEG): container finished" podID="1fc83150-c87b-4013-b6f4-f23f331d62c9" containerID="72937019c32d6cbe160b397f4688e8bb8df26897d7ed0afb72c791bc38b85850" exitCode=1
Sep 08 13:19:00.368633 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:00.368326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd" event={"ID":"1fc83150-c87b-4013-b6f4-f23f331d62c9","Type":"ContainerDied","Data":"72937019c32d6cbe160b397f4688e8bb8df26897d7ed0afb72c791bc38b85850"}
Sep 08 13:19:00.368723 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:00.368646    2581 scope.go:117] "RemoveContainer" containerID="72937019c32d6cbe160b397f4688e8bb8df26897d7ed0afb72c791bc38b85850"
Sep 08 13:19:00.369854 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:00.369824    2581 generic.go:356] "Generic (PLEG): container finished" podID="c3c969ab-d328-4c83-99ce-8864535b2bc2" containerID="a4afcb9a1901bccd7469565e11b284a4cef325ae060b1e36da147f944d43b742" exitCode=255
Sep 08 13:19:00.369933 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:00.369857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp" event={"ID":"c3c969ab-d328-4c83-99ce-8864535b2bc2","Type":"ContainerDied","Data":"a4afcb9a1901bccd7469565e11b284a4cef325ae060b1e36da147f944d43b742"}
Sep 08 13:19:00.370171 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:00.370154    2581 scope.go:117] "RemoveContainer" containerID="a4afcb9a1901bccd7469565e11b284a4cef325ae060b1e36da147f944d43b742"
Sep 08 13:19:01.005958 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:01.005925    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:19:01.375064 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:01.375022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd" event={"ID":"1fc83150-c87b-4013-b6f4-f23f331d62c9","Type":"ContainerStarted","Data":"271c94b7e44fc5b27d0c1554a69c3b142c02758972db6814570ad8782918f3c0"}
Sep 08 13:19:01.375528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:01.375240    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:19:01.375884 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:01.375855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5794954d67-l9czd"
Sep 08 13:19:01.377027 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:01.377008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6fd68b96c8-q2dmp" event={"ID":"c3c969ab-d328-4c83-99ce-8864535b2bc2","Type":"ContainerStarted","Data":"6e0e0d9b931321caaf69199b1d62fdd81f8c7a9a08e7cc9cf3057c8a12c88b5d"}
Sep 08 13:19:01.817306 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:01.817281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:19:21.055076 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.055049    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zn476"]
Sep 08 13:19:21.058194 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.058175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.061368 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.061334    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:19:21.061469 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.061405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kdgsm\""
Sep 08 13:19:21.061870 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.061854    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:19:21.061927 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.061854    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:19:21.067022 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.065525    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:19:21.071901 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.071879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zn476"]
Sep 08 13:19:21.177362 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.177323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/765018f8-64fc-4b09-9a08-a9e65fb049f0-crio-socket\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.177538 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.177376    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.177538 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.177398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.177538 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.177462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrtw8\" (UniqueName: \"kubernetes.io/projected/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-api-access-xrtw8\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.177538 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.177506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/765018f8-64fc-4b09-9a08-a9e65fb049f0-data-volume\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.278832 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.278803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.278984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.278838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.278984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.278867    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrtw8\" (UniqueName: \"kubernetes.io/projected/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-api-access-xrtw8\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.278984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.278895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/765018f8-64fc-4b09-9a08-a9e65fb049f0-data-volume\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.278984 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:21.278941    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:21.278984 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:21.278970    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:21.278984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.278977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/765018f8-64fc-4b09-9a08-a9e65fb049f0-crio-socket\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.279216 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:21.279017    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:21.778997345 +0000 UTC m=+191.482783187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:21.279216 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:21.279033    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:21.779026659 +0000 UTC m=+191.482812485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:21.279216 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.279064    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/765018f8-64fc-4b09-9a08-a9e65fb049f0-crio-socket\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.279316 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.279214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/765018f8-64fc-4b09-9a08-a9e65fb049f0-data-volume\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.292787 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.292762    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrtw8\" (UniqueName: \"kubernetes.io/projected/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-api-access-xrtw8\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.784721 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.784687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.784886 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:21.784726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:21.784886 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:21.784821    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:21.784886 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:21.784825    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:21.784886 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:21.784881    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:22.78486778 +0000 UTC m=+192.488653607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:21.785034 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:21.784894    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:22.784889026 +0000 UTC m=+192.488674852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:22.793125 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:22.793092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:22.793125 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:22.793127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:22.793495 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:22.793202    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:22.793495 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:22.793238    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:22.793495 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:22.793275    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:24.793260528 +0000 UTC m=+194.497046355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:22.793495 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:22.793293    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:24.793281091 +0000 UTC m=+194.497066921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:22.805072 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:22.805051    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxllr_2f80b4d3-387c-4580-9899-0be849014b1e/dns-node-resolver/0.log"
Sep 08 13:19:24.205999 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:24.205972    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lnvw8_d3fd4abc-ff7d-4512-a9f2-8103510ca5c6/node-ca/0.log"
Sep 08 13:19:24.806704 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:24.806654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:24.806887 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:24.806713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:24.806887 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:24.806822    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:24.806887 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:24.806873    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:28.806861053 +0000 UTC m=+198.510646892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:24.807029 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:24.806822    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:24.807029 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:24.806942    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:28.806930523 +0000 UTC m=+198.510716366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:24.807991 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:24.807962    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qx8ch_a1e7cc14-c9c0-4576-931e-53bc31b9326b/serve-healthcheck-canary/0.log"
Sep 08 13:19:28.839783 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:28.839746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:28.839783 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:28.839787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:28.840362 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:28.839861    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:28.840362 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:28.839894    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:28.840362 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:28.839926    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:36.839905328 +0000 UTC m=+206.543691160 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:28.840362 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:28.839947    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:36.839934166 +0000 UTC m=+206.543719992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:36.898568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:36.898531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:36.898568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:36.898572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:36.899000 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:36.898697    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:36.899000 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:19:36.898783    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls podName:765018f8-64fc-4b09-9a08-a9e65fb049f0 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:52.89875438 +0000 UTC m=+222.602540211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zn476" (UID: "765018f8-64fc-4b09-9a08-a9e65fb049f0") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:36.899076 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:36.899052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/765018f8-64fc-4b09-9a08-a9e65fb049f0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:37.228774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:37.228666    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" podUID="f31a1fe3-0ae3-41d3-81a0-926e5a2f7510" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:19:47.228741 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:47.228705    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" podUID="f31a1fe3-0ae3-41d3-81a0-926e5a2f7510" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:19:52.908907 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:52.908868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:52.911211 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:52.911191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/765018f8-64fc-4b09-9a08-a9e65fb049f0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zn476\" (UID: \"765018f8-64fc-4b09-9a08-a9e65fb049f0\") " pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:53.169366 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:53.169277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zn476"
Sep 08 13:19:53.441167 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:53.441091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zn476"]
Sep 08 13:19:53.444267 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:19:53.444238    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765018f8_64fc_4b09_9a08_a9e65fb049f0.slice/crio-5196e1134bcd0e963b6a9404ab9700ca03598d20cc33810ecbc38d55a182b427 WatchSource:0}: Error finding container 5196e1134bcd0e963b6a9404ab9700ca03598d20cc33810ecbc38d55a182b427: Status 404 returned error can't find the container with id 5196e1134bcd0e963b6a9404ab9700ca03598d20cc33810ecbc38d55a182b427
Sep 08 13:19:53.526690 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:53.526633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zn476" event={"ID":"765018f8-64fc-4b09-9a08-a9e65fb049f0","Type":"ContainerStarted","Data":"91841a92c541cb664fb664d1b234ae4b52ac1a4c73ed009fb823347c460561fd"}
Sep 08 13:19:53.526817 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:53.526715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zn476" event={"ID":"765018f8-64fc-4b09-9a08-a9e65fb049f0","Type":"ContainerStarted","Data":"5196e1134bcd0e963b6a9404ab9700ca03598d20cc33810ecbc38d55a182b427"}
Sep 08 13:19:55.534820 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:55.534726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zn476" event={"ID":"765018f8-64fc-4b09-9a08-a9e65fb049f0","Type":"ContainerStarted","Data":"26da88a298fa1d36e4ea6f691f972ac2eb1c033531f93d8d258af06b2d66b84f"}
Sep 08 13:19:57.228874 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:57.228831    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" podUID="f31a1fe3-0ae3-41d3-81a0-926e5a2f7510" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 08 13:19:57.229306 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:57.228907    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg"
Sep 08 13:19:57.229483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:57.229448    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b81ca696fb5bf40fbb4a764a48c0fcca5a91402a9e193fe065a971d64308b74f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 08 13:19:57.229539 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:57.229514    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" podUID="f31a1fe3-0ae3-41d3-81a0-926e5a2f7510" containerName="service-proxy" containerID="cri-o://b81ca696fb5bf40fbb4a764a48c0fcca5a91402a9e193fe065a971d64308b74f" gracePeriod=30
Sep 08 13:19:57.544195 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:57.544162    2581 generic.go:356] "Generic (PLEG): container finished" podID="f31a1fe3-0ae3-41d3-81a0-926e5a2f7510" containerID="b81ca696fb5bf40fbb4a764a48c0fcca5a91402a9e193fe065a971d64308b74f" exitCode=2
Sep 08 13:19:57.544375 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:57.544231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" event={"ID":"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510","Type":"ContainerDied","Data":"b81ca696fb5bf40fbb4a764a48c0fcca5a91402a9e193fe065a971d64308b74f"}
Sep 08 13:19:58.549174 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:58.549131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zn476" event={"ID":"765018f8-64fc-4b09-9a08-a9e65fb049f0","Type":"ContainerStarted","Data":"2e7ebbcd89b3b11a79de8dfd033ac04532fabcd8e6a3dbafbed10a09fd1a3bcd"}
Sep 08 13:19:58.551485 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:58.551458    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-69d685bffc-znwsg" event={"ID":"f31a1fe3-0ae3-41d3-81a0-926e5a2f7510","Type":"ContainerStarted","Data":"25a9b9faa179c3e47891457eeaee857c2d028deea36b4320ca4a4faf2554ee1b"}
Sep 08 13:19:58.569282 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:19:58.569235    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zn476" podStartSLOduration=33.450178581 podStartE2EDuration="37.569224112s" totalImagesPullingTime="4.119045531s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:19:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:58.56762238 +0000 UTC m=+228.271408228" watchObservedRunningTime="2026-09-08 13:19:58.569224112 +0000 UTC m=+228.273009961"
Sep 08 13:20:05.317205 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.317171    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dd9rn"]
Sep 08 13:20:05.320373 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.320353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.322538 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.322508    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:20:05.322785 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.322763    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:20:05.322898 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.322810    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:20:05.322898 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.322850    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:20:05.322898 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.322856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vhknt\""
Sep 08 13:20:05.322898 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.322850    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:20:05.323372 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.323354    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:20:05.398285 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398249    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dd71c4-df9d-44a4-89f4-030403050ffc-metrics-client-ca\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.398285 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-textfile\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.398439 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-wtmp\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.398439 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lftxv\" (UniqueName: \"kubernetes.io/projected/f7dd71c4-df9d-44a4-89f4-030403050ffc-kube-api-access-lftxv\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.398504 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-accelerators-collector-config\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.398504 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-tls\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.398504 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-root\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.398614 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.398614 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.398545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-sys\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499479 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dd71c4-df9d-44a4-89f4-030403050ffc-metrics-client-ca\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499585 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-textfile\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499585 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-wtmp\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499585 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lftxv\" (UniqueName: \"kubernetes.io/projected/f7dd71c4-df9d-44a4-89f4-030403050ffc-kube-api-access-lftxv\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499772 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-accelerators-collector-config\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499772 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-tls\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499772 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-root\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499772 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499772 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-sys\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499772 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499752    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-wtmp\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499994 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499809    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-root\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499994 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7dd71c4-df9d-44a4-89f4-030403050ffc-sys\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.499994 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.499852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-textfile\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.500171 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.500149    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7dd71c4-df9d-44a4-89f4-030403050ffc-metrics-client-ca\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.500373 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.500356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-accelerators-collector-config\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.502067 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.502051    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-tls\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.502199 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.502181    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7dd71c4-df9d-44a4-89f4-030403050ffc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.508051 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.508028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lftxv\" (UniqueName: \"kubernetes.io/projected/f7dd71c4-df9d-44a4-89f4-030403050ffc-kube-api-access-lftxv\") pod \"node-exporter-dd9rn\" (UID: \"f7dd71c4-df9d-44a4-89f4-030403050ffc\") " pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.628812 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:05.628755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dd9rn"
Sep 08 13:20:05.638408 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:20:05.638380    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dd71c4_df9d_44a4_89f4_030403050ffc.slice/crio-666c9323753fffaa9c71c608b88a935f59c23da4812b305652bbccfbe760206c WatchSource:0}: Error finding container 666c9323753fffaa9c71c608b88a935f59c23da4812b305652bbccfbe760206c: Status 404 returned error can't find the container with id 666c9323753fffaa9c71c608b88a935f59c23da4812b305652bbccfbe760206c
Sep 08 13:20:06.580760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:06.580725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dd9rn" event={"ID":"f7dd71c4-df9d-44a4-89f4-030403050ffc","Type":"ContainerStarted","Data":"666c9323753fffaa9c71c608b88a935f59c23da4812b305652bbccfbe760206c"}
Sep 08 13:20:07.585571 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:07.585535    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7dd71c4-df9d-44a4-89f4-030403050ffc" containerID="51f6d1dbab4d6920dd565fc0cad0140179af3a42ef696e5ed600a257c06b9b38" exitCode=0
Sep 08 13:20:07.585571 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:07.585572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dd9rn" event={"ID":"f7dd71c4-df9d-44a4-89f4-030403050ffc","Type":"ContainerDied","Data":"51f6d1dbab4d6920dd565fc0cad0140179af3a42ef696e5ed600a257c06b9b38"}
Sep 08 13:20:07.906761 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:07.906688    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687dc7f5fd-c4fhz"]
Sep 08 13:20:07.906889 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:20:07.906846    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz" podUID="86c78164-13cb-4515-a395-08a81ccc9594"
Sep 08 13:20:08.590492 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.590463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:20:08.590909 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.590490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dd9rn" event={"ID":"f7dd71c4-df9d-44a4-89f4-030403050ffc","Type":"ContainerStarted","Data":"280b2fc8366f0740e7e03139dd694894afa618428ea9adcc272893b24bd5d2fb"}
Sep 08 13:20:08.590909 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.590552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dd9rn" event={"ID":"f7dd71c4-df9d-44a4-89f4-030403050ffc","Type":"ContainerStarted","Data":"9fa7c7cf554e7034e5800b13cf832e67d9663753f16221029d71328b31d27655"}
Sep 08 13:20:08.596578 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.596556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:20:08.618399 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.618355    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dd9rn" podStartSLOduration=2.286496564 podStartE2EDuration="3.618339528s" totalImagesPullingTime="1.264905077s" totalInitContainerRuntime="66.937887ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:08.616529749 +0000 UTC m=+238.320315598" watchObservedRunningTime="2026-09-08 13:20:08.618339528 +0000 UTC m=+238.322125391"
Sep 08 13:20:08.625477 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-bound-sa-token\") pod \"86c78164-13cb-4515-a395-08a81ccc9594\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") "
Sep 08 13:20:08.625573 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5w7m\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-kube-api-access-h5w7m\") pod \"86c78164-13cb-4515-a395-08a81ccc9594\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") "
Sep 08 13:20:08.625573 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86c78164-13cb-4515-a395-08a81ccc9594-ca-trust-extracted\") pod \"86c78164-13cb-4515-a395-08a81ccc9594\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") "
Sep 08 13:20:08.625573 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625547    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-registry-certificates\") pod \"86c78164-13cb-4515-a395-08a81ccc9594\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") "
Sep 08 13:20:08.625725 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-installation-pull-secrets\") pod \"86c78164-13cb-4515-a395-08a81ccc9594\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") "
Sep 08 13:20:08.625725 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625601    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-trusted-ca\") pod \"86c78164-13cb-4515-a395-08a81ccc9594\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") "
Sep 08 13:20:08.625725 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-image-registry-private-configuration\") pod \"86c78164-13cb-4515-a395-08a81ccc9594\" (UID: \"86c78164-13cb-4515-a395-08a81ccc9594\") "
Sep 08 13:20:08.625909 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625880    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c78164-13cb-4515-a395-08a81ccc9594-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "86c78164-13cb-4515-a395-08a81ccc9594" (UID: "86c78164-13cb-4515-a395-08a81ccc9594"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:20:08.625976 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.625951    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "86c78164-13cb-4515-a395-08a81ccc9594" (UID: "86c78164-13cb-4515-a395-08a81ccc9594"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:08.626044 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.626019    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "86c78164-13cb-4515-a395-08a81ccc9594" (UID: "86c78164-13cb-4515-a395-08a81ccc9594"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:20:08.628200 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.628171    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "86c78164-13cb-4515-a395-08a81ccc9594" (UID: "86c78164-13cb-4515-a395-08a81ccc9594"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:20:08.628308 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.628208    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "86c78164-13cb-4515-a395-08a81ccc9594" (UID: "86c78164-13cb-4515-a395-08a81ccc9594"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:20:08.628308 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.628249    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-kube-api-access-h5w7m" (OuterVolumeSpecName: "kube-api-access-h5w7m") pod "86c78164-13cb-4515-a395-08a81ccc9594" (UID: "86c78164-13cb-4515-a395-08a81ccc9594"). InnerVolumeSpecName "kube-api-access-h5w7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:20:08.628308 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.628290    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "86c78164-13cb-4515-a395-08a81ccc9594" (UID: "86c78164-13cb-4515-a395-08a81ccc9594"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:20:08.726619 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.726590    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-bound-sa-token\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:20:08.726619 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.726617    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5w7m\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-kube-api-access-h5w7m\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:20:08.726619 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.726627    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/86c78164-13cb-4515-a395-08a81ccc9594-ca-trust-extracted\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:20:08.726799 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.726636    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-registry-certificates\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:20:08.726799 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.726644    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-installation-pull-secrets\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:20:08.726799 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.726652    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/86c78164-13cb-4515-a395-08a81ccc9594-trusted-ca\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:20:08.726799 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:08.726660    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/86c78164-13cb-4515-a395-08a81ccc9594-image-registry-private-configuration\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:20:09.593167 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:09.593140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-687dc7f5fd-c4fhz"
Sep 08 13:20:09.624561 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:09.624535    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-687dc7f5fd-c4fhz"]
Sep 08 13:20:09.630715 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:09.630691    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-687dc7f5fd-c4fhz"]
Sep 08 13:20:09.734527 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:09.734504    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/86c78164-13cb-4515-a395-08a81ccc9594-registry-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:20:10.821181 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:10.821149    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86c78164-13cb-4515-a395-08a81ccc9594" path="/var/lib/kubelet/pods/86c78164-13cb-4515-a395-08a81ccc9594/volumes"
Sep 08 13:20:22.613568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:22.613531    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:20:22.615837 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:22.615806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5ec803-ab0d-4166-bf48-77a72a26b284-metrics-certs\") pod \"network-metrics-daemon-mpqn2\" (UID: \"1a5ec803-ab0d-4166-bf48-77a72a26b284\") " pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:20:22.820950 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:22.820925    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2p8w4\""
Sep 08 13:20:22.829042 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:22.829019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpqn2"
Sep 08 13:20:23.132171 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:23.132148    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mpqn2"]
Sep 08 13:20:23.134355 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:20:23.134327    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5ec803_ab0d_4166_bf48_77a72a26b284.slice/crio-db553abbdf7638f28e1da9857082e6b621ab73e9601a9f9a18f736d7f1825cf9 WatchSource:0}: Error finding container db553abbdf7638f28e1da9857082e6b621ab73e9601a9f9a18f736d7f1825cf9: Status 404 returned error can't find the container with id db553abbdf7638f28e1da9857082e6b621ab73e9601a9f9a18f736d7f1825cf9
Sep 08 13:20:23.638195 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:23.638157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpqn2" event={"ID":"1a5ec803-ab0d-4166-bf48-77a72a26b284","Type":"ContainerStarted","Data":"db553abbdf7638f28e1da9857082e6b621ab73e9601a9f9a18f736d7f1825cf9"}
Sep 08 13:20:28.655312 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:28.655276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpqn2" event={"ID":"1a5ec803-ab0d-4166-bf48-77a72a26b284","Type":"ContainerStarted","Data":"edd051c2d3c8b5832a2531aba29cb16314ecddf36349c0686bfc5dcfe03b3429"}
Sep 08 13:20:28.655722 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:28.655336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpqn2" event={"ID":"1a5ec803-ab0d-4166-bf48-77a72a26b284","Type":"ContainerStarted","Data":"b1b90cce8441b116f819377b75e1dc86b1a454c8a81b8c588a81dca75a912f7d"}
Sep 08 13:20:28.674111 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:28.674032    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mpqn2" podStartSLOduration=253.069734915 podStartE2EDuration="4m17.674018738s" totalImagesPullingTime="4.604283823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:28.67381201 +0000 UTC m=+258.377597873" watchObservedRunningTime="2026-09-08 13:20:28.674018738 +0000 UTC m=+258.377804587"
Sep 08 13:20:41.719752 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:41.719722    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxllr_2f80b4d3-387c-4580-9899-0be849014b1e/dns-node-resolver/0.log"
Sep 08 13:20:51.330730 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:20:51.330665    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ntxdb" podUID="c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810"
Sep 08 13:20:51.730856 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:51.730782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:20:55.137496 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:55.137455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:20:55.139901 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:55.139881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810-metrics-tls\") pod \"dns-default-ntxdb\" (UID: \"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810\") " pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:20:55.333638 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:55.333610    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8sm9m\""
Sep 08 13:20:55.341882 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:55.341861    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:20:55.614693 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:55.614650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ntxdb"]
Sep 08 13:20:55.744280 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:55.744203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ntxdb" event={"ID":"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810","Type":"ContainerStarted","Data":"be0dcbcbe8edf4241cd4f9942c4ba34c9183b107ae81b832b6c7980b5a40a9f9"}
Sep 08 13:20:57.752976 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:57.752942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ntxdb" event={"ID":"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810","Type":"ContainerStarted","Data":"c8daca96f33d911f90372f14167eb3be7fcc7a07455aab2ea40698471e0d6525"}
Sep 08 13:20:57.753341 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:57.753018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ntxdb" event={"ID":"c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810","Type":"ContainerStarted","Data":"44a6fe13a4fe9b3744b73462280bfee295fe45c1e119fa594a52b1d132b23aa0"}
Sep 08 13:20:57.753341 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:57.753106    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:20:57.773849 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:20:57.773803    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ntxdb" podStartSLOduration=251.263267804 podStartE2EDuration="4m12.773788637s" totalImagesPullingTime="1.510520833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:57.772454104 +0000 UTC m=+287.476239973" watchObservedRunningTime="2026-09-08 13:20:57.773788637 +0000 UTC m=+287.477574525"
Sep 08 13:21:07.759399 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:21:07.759367    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ntxdb"
Sep 08 13:21:10.721836 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:21:10.721808    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:24:00.332015 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.331979    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d4bml"]
Sep 08 13:24:00.334970 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.334952    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:00.337191 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.337170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-2g6xl\""
Sep 08 13:24:00.337431 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.337412    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 08 13:24:00.337922 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.337904    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 08 13:24:00.338101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.338081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 08 13:24:00.338210 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.338117    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 08 13:24:00.338210 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.338121    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 08 13:24:00.345890 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.345865    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d4bml"]
Sep 08 13:24:00.436090 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.436057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:00.436216 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.436122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmp6\" (UniqueName: \"kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-kube-api-access-2jmp6\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:00.436216 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.436176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-cabundle0\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:00.537463 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.537434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-cabundle0\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:00.537576 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.537486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:00.537576 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.537504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmp6\" (UniqueName: \"kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-kube-api-access-2jmp6\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:00.537660 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:24:00.537596    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 08 13:24:00.537660 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:24:00.537614    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 08 13:24:00.537660 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:24:00.537622    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-d4bml: references non-existent secret key: ca.crt
Sep 08 13:24:00.537799 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:24:00.537696    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates podName:ca16d762-c23e-4bed-905f-aa08c4ad7ed3 nodeName:}" failed. No retries permitted until 2026-09-08 13:24:01.037660584 +0000 UTC m=+470.741446410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates") pod "keda-operator-579c8958f7-d4bml" (UID: "ca16d762-c23e-4bed-905f-aa08c4ad7ed3") : references non-existent secret key: ca.crt
Sep 08 13:24:00.538091 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.538073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-cabundle0\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:00.549765 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:00.549744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmp6\" (UniqueName: \"kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-kube-api-access-2jmp6\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:01.040022 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.039993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:01.040179 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:24:01.040125    2581 secret.go:281] references non-existent secret key: ca.crt
Sep 08 13:24:01.040179 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:24:01.040137    2581 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 08 13:24:01.040179 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:24:01.040145    2581 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-d4bml: references non-existent secret key: ca.crt
Sep 08 13:24:01.040273 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:24:01.040203    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates podName:ca16d762-c23e-4bed-905f-aa08c4ad7ed3 nodeName:}" failed. No retries permitted until 2026-09-08 13:24:02.040187368 +0000 UTC m=+471.743973198 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates") pod "keda-operator-579c8958f7-d4bml" (UID: "ca16d762-c23e-4bed-905f-aa08c4ad7ed3") : references non-existent secret key: ca.crt
Sep 08 13:24:01.358454 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.358373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jgclb"]
Sep 08 13:24:01.361502 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.361487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:01.364040 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.364015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 08 13:24:01.371553 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.371531    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jgclb"]
Sep 08 13:24:01.442919 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.442890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a128718f-10ed-4b73-82de-c35a96cec310-certificates\") pod \"keda-admission-6c4d864b86-jgclb\" (UID: \"a128718f-10ed-4b73-82de-c35a96cec310\") " pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:01.442919 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.442918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbwdv\" (UniqueName: \"kubernetes.io/projected/a128718f-10ed-4b73-82de-c35a96cec310-kube-api-access-zbwdv\") pod \"keda-admission-6c4d864b86-jgclb\" (UID: \"a128718f-10ed-4b73-82de-c35a96cec310\") " pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:01.544130 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.544097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a128718f-10ed-4b73-82de-c35a96cec310-certificates\") pod \"keda-admission-6c4d864b86-jgclb\" (UID: \"a128718f-10ed-4b73-82de-c35a96cec310\") " pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:01.544130 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.544130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbwdv\" (UniqueName: \"kubernetes.io/projected/a128718f-10ed-4b73-82de-c35a96cec310-kube-api-access-zbwdv\") pod \"keda-admission-6c4d864b86-jgclb\" (UID: \"a128718f-10ed-4b73-82de-c35a96cec310\") " pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:01.546653 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.546634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a128718f-10ed-4b73-82de-c35a96cec310-certificates\") pod \"keda-admission-6c4d864b86-jgclb\" (UID: \"a128718f-10ed-4b73-82de-c35a96cec310\") " pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:01.554903 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.554880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbwdv\" (UniqueName: \"kubernetes.io/projected/a128718f-10ed-4b73-82de-c35a96cec310-kube-api-access-zbwdv\") pod \"keda-admission-6c4d864b86-jgclb\" (UID: \"a128718f-10ed-4b73-82de-c35a96cec310\") " pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:01.671232 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.671178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:01.941966 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.941905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jgclb"]
Sep 08 13:24:01.944828 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:24:01.944797    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda128718f_10ed_4b73_82de_c35a96cec310.slice/crio-7c96c089ac0b250989865a9d7f03b8bdf905638d07a4f8943f310d01158b2815 WatchSource:0}: Error finding container 7c96c089ac0b250989865a9d7f03b8bdf905638d07a4f8943f310d01158b2815: Status 404 returned error can't find the container with id 7c96c089ac0b250989865a9d7f03b8bdf905638d07a4f8943f310d01158b2815
Sep 08 13:24:01.946300 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:01.946279    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:24:02.046716 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:02.046690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:02.049149 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:02.049120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ca16d762-c23e-4bed-905f-aa08c4ad7ed3-certificates\") pod \"keda-operator-579c8958f7-d4bml\" (UID: \"ca16d762-c23e-4bed-905f-aa08c4ad7ed3\") " pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:02.144387 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:02.144367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:02.342064 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:02.342036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jgclb" event={"ID":"a128718f-10ed-4b73-82de-c35a96cec310","Type":"ContainerStarted","Data":"7c96c089ac0b250989865a9d7f03b8bdf905638d07a4f8943f310d01158b2815"}
Sep 08 13:24:02.416348 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:02.416290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-d4bml"]
Sep 08 13:24:02.419360 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:24:02.419334    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca16d762_c23e_4bed_905f_aa08c4ad7ed3.slice/crio-fd751983724f1a2873935e0c71204d2d95b09a988fd42e7f54430e3c36978028 WatchSource:0}: Error finding container fd751983724f1a2873935e0c71204d2d95b09a988fd42e7f54430e3c36978028: Status 404 returned error can't find the container with id fd751983724f1a2873935e0c71204d2d95b09a988fd42e7f54430e3c36978028
Sep 08 13:24:03.346204 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:03.346171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-d4bml" event={"ID":"ca16d762-c23e-4bed-905f-aa08c4ad7ed3","Type":"ContainerStarted","Data":"fd751983724f1a2873935e0c71204d2d95b09a988fd42e7f54430e3c36978028"}
Sep 08 13:24:07.361274 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:07.361236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jgclb" event={"ID":"a128718f-10ed-4b73-82de-c35a96cec310","Type":"ContainerStarted","Data":"87db85155e13fe9ddea770049c8aea25c13a8cf9644646b6161120b008ab3ff1"}
Sep 08 13:24:07.361718 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:07.361383    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:24:07.362804 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:07.362773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-d4bml" event={"ID":"ca16d762-c23e-4bed-905f-aa08c4ad7ed3","Type":"ContainerStarted","Data":"59cf2f2b4fa860171a84b334b260fe4c4dff6355ad831eb17b3d43833149245e"}
Sep 08 13:24:07.362911 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:07.362825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:07.384983 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:07.384938    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-jgclb" podStartSLOduration=1.167447513 podStartE2EDuration="6.384925574s" totalImagesPullingTime="5.217478061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:07.383219282 +0000 UTC m=+477.087005130" watchObservedRunningTime="2026-09-08 13:24:07.384925574 +0000 UTC m=+477.088711431"
Sep 08 13:24:07.402390 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:07.402350    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-d4bml" podStartSLOduration=3.477832911 podStartE2EDuration="7.402338932s" totalImagesPullingTime="3.924506021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:07.400812054 +0000 UTC m=+477.104597904" watchObservedRunningTime="2026-09-08 13:24:07.402338932 +0000 UTC m=+477.106124803"
Sep 08 13:24:28.368784 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:28.368753    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-d4bml"
Sep 08 13:24:28.369277 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:24:28.369102    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-jgclb"
Sep 08 13:25:21.629480 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.629401    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"]
Sep 08 13:25:21.632486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.632468    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:21.634925 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.634900    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 08 13:25:21.634925 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.634915    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 08 13:25:21.635112 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.634960    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-k26lg\""
Sep 08 13:25:21.635926 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.635907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 08 13:25:21.642007 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.641980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"]
Sep 08 13:25:21.664698 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.664656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60b6a7a2-eb8d-42ed-9863-493653caf289-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-bv94t\" (UID: \"60b6a7a2-eb8d-42ed-9863-493653caf289\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:21.664821 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.664758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m786h\" (UniqueName: \"kubernetes.io/projected/60b6a7a2-eb8d-42ed-9863-493653caf289-kube-api-access-m786h\") pod \"llmisvc-controller-manager-6f9bbc7b7c-bv94t\" (UID: \"60b6a7a2-eb8d-42ed-9863-493653caf289\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:21.765426 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.765402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60b6a7a2-eb8d-42ed-9863-493653caf289-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-bv94t\" (UID: \"60b6a7a2-eb8d-42ed-9863-493653caf289\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:21.765527 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.765433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m786h\" (UniqueName: \"kubernetes.io/projected/60b6a7a2-eb8d-42ed-9863-493653caf289-kube-api-access-m786h\") pod \"llmisvc-controller-manager-6f9bbc7b7c-bv94t\" (UID: \"60b6a7a2-eb8d-42ed-9863-493653caf289\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:21.765594 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:25:21.765557    2581 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 08 13:25:21.765664 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:25:21.765643    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b6a7a2-eb8d-42ed-9863-493653caf289-cert podName:60b6a7a2-eb8d-42ed-9863-493653caf289 nodeName:}" failed. No retries permitted until 2026-09-08 13:25:22.26562323 +0000 UTC m=+551.969409077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/60b6a7a2-eb8d-42ed-9863-493653caf289-cert") pod "llmisvc-controller-manager-6f9bbc7b7c-bv94t" (UID: "60b6a7a2-eb8d-42ed-9863-493653caf289") : secret "llmisvc-webhook-server-cert" not found
Sep 08 13:25:21.774900 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:21.774873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m786h\" (UniqueName: \"kubernetes.io/projected/60b6a7a2-eb8d-42ed-9863-493653caf289-kube-api-access-m786h\") pod \"llmisvc-controller-manager-6f9bbc7b7c-bv94t\" (UID: \"60b6a7a2-eb8d-42ed-9863-493653caf289\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:22.268203 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:22.268169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60b6a7a2-eb8d-42ed-9863-493653caf289-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-bv94t\" (UID: \"60b6a7a2-eb8d-42ed-9863-493653caf289\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:22.270568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:22.270539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60b6a7a2-eb8d-42ed-9863-493653caf289-cert\") pod \"llmisvc-controller-manager-6f9bbc7b7c-bv94t\" (UID: \"60b6a7a2-eb8d-42ed-9863-493653caf289\") " pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:22.542948 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:22.542918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:22.813015 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:22.812946    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"]
Sep 08 13:25:22.816205 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:25:22.816172    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod60b6a7a2_eb8d_42ed_9863_493653caf289.slice/crio-a397a9571a674da0e611ce6419736d904fb665eae69eceea764da804bd2b3480 WatchSource:0}: Error finding container a397a9571a674da0e611ce6419736d904fb665eae69eceea764da804bd2b3480: Status 404 returned error can't find the container with id a397a9571a674da0e611ce6419736d904fb665eae69eceea764da804bd2b3480
Sep 08 13:25:23.614870 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:23.614828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t" event={"ID":"60b6a7a2-eb8d-42ed-9863-493653caf289","Type":"ContainerStarted","Data":"a397a9571a674da0e611ce6419736d904fb665eae69eceea764da804bd2b3480"}
Sep 08 13:25:26.625928 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:26.625895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t" event={"ID":"60b6a7a2-eb8d-42ed-9863-493653caf289","Type":"ContainerStarted","Data":"9b6668a685950f1f2d06329b04e0af2cfd2b933d38158026f3b34e4105af89cf"}
Sep 08 13:25:26.626291 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:26.626025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:25:26.651642 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:26.651587    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t" podStartSLOduration=2.885091872 podStartE2EDuration="5.651570397s" totalImagesPullingTime="2.766478525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:25:26.650564318 +0000 UTC m=+556.354350179" watchObservedRunningTime="2026-09-08 13:25:26.651570397 +0000 UTC m=+556.355356247"
Sep 08 13:25:57.632545 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:25:57.632512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6f9bbc7b7c-bv94t"
Sep 08 13:28:28.251225 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.251153    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"]
Sep 08 13:28:28.254294 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.254278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.256773 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.256745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:28:28.256893 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.256839    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 08 13:28:28.256959 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.256937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:28:28.257022 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.256995    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 08 13:28:28.257609 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.257593    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:28:28.257724 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.257609    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:28:28.258542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.258522    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:28:28.270640 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.270620    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"]
Sep 08 13:28:28.378145 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.378116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2015d03f-00cc-4908-97ba-c871c40fe283-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.378145 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.378147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nzr\" (UniqueName: \"kubernetes.io/projected/2015d03f-00cc-4908-97ba-c871c40fe283-kube-api-access-89nzr\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.378320 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.378195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.378320 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.378285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2015d03f-00cc-4908-97ba-c871c40fe283-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.378320 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.378309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.478940 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.478912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.479045 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.478954    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2015d03f-00cc-4908-97ba-c871c40fe283-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.479045 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.478972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.479120 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.479090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2015d03f-00cc-4908-97ba-c871c40fe283-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.479188 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.479120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nzr\" (UniqueName: \"kubernetes.io/projected/2015d03f-00cc-4908-97ba-c871c40fe283-kube-api-access-89nzr\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.479331 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.479309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2015d03f-00cc-4908-97ba-c871c40fe283-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.479643 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.479624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.479737 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.479646    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.481573 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.481550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2015d03f-00cc-4908-97ba-c871c40fe283-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.487952 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.487927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nzr\" (UniqueName: \"kubernetes.io/projected/2015d03f-00cc-4908-97ba-c871c40fe283-kube-api-access-89nzr\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-5v7rn\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.564897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.564867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:28.835402 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:28.835332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"]
Sep 08 13:28:28.838112 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:28:28.838088    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2015d03f_00cc_4908_97ba_c871c40fe283.slice/crio-b8abba28a452b78d78a7e32f9cfa0af7893f0ab01056c7df7042484803c45a84 WatchSource:0}: Error finding container b8abba28a452b78d78a7e32f9cfa0af7893f0ab01056c7df7042484803c45a84: Status 404 returned error can't find the container with id b8abba28a452b78d78a7e32f9cfa0af7893f0ab01056c7df7042484803c45a84
Sep 08 13:28:29.231114 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:29.231028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" event={"ID":"2015d03f-00cc-4908-97ba-c871c40fe283","Type":"ContainerStarted","Data":"b8abba28a452b78d78a7e32f9cfa0af7893f0ab01056c7df7042484803c45a84"}
Sep 08 13:28:35.255079 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:35.255037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" event={"ID":"2015d03f-00cc-4908-97ba-c871c40fe283","Type":"ContainerStarted","Data":"81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161"}
Sep 08 13:28:39.271438 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:39.271341    2581 generic.go:356] "Generic (PLEG): container finished" podID="2015d03f-00cc-4908-97ba-c871c40fe283" containerID="81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161" exitCode=0
Sep 08 13:28:39.271438 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:39.271419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" event={"ID":"2015d03f-00cc-4908-97ba-c871c40fe283","Type":"ContainerDied","Data":"81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161"}
Sep 08 13:28:56.343428 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:56.343385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" event={"ID":"2015d03f-00cc-4908-97ba-c871c40fe283","Type":"ContainerStarted","Data":"54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98"}
Sep 08 13:28:58.352775 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:58.352741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" event={"ID":"2015d03f-00cc-4908-97ba-c871c40fe283","Type":"ContainerStarted","Data":"4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd"}
Sep 08 13:28:58.353176 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:58.352872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:58.377803 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:58.377758    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podStartSLOduration=1.928822778 podStartE2EDuration="30.377744648s" totalImagesPullingTime="24.785960974s" totalInitContainerRuntime="3.662960896s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:28:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:28:58.375729308 +0000 UTC m=+768.079515168" watchObservedRunningTime="2026-09-08 13:28:58.377744648 +0000 UTC m=+768.081530497"
Sep 08 13:28:59.356581 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:59.356546    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:28:59.357747 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:28:59.357718    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:29:00.360029 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:29:00.359990    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:29:05.364896 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:29:05.364863    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:29:05.365471 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:29:05.365442    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:29:15.365899 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:29:15.365855    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:29:25.366090 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:29:25.366053    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:29:35.365953 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:29:35.365873    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:29:45.366216 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:29:45.366179    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:29:55.366284 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:29:55.366244    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:30:05.365774 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:05.365734    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.16:8080: connect: connection refused"
Sep 08 13:30:12.151690 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:12.151645    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"]
Sep 08 13:30:12.152206 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:12.151882    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container" containerID="cri-o://54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98" gracePeriod=30
Sep 08 13:30:12.152206 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:12.151973    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kube-rbac-proxy" containerID="cri-o://4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd" gracePeriod=30
Sep 08 13:30:12.152816 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:12.152797    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:30:12.613912 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:12.613876    2581 generic.go:356] "Generic (PLEG): container finished" podID="2015d03f-00cc-4908-97ba-c871c40fe283" containerID="4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd" exitCode=2
Sep 08 13:30:12.614068 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:12.613948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" event={"ID":"2015d03f-00cc-4908-97ba-c871c40fe283","Type":"ContainerDied","Data":"4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd"}
Sep 08 13:30:15.360836 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:15.360800    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.16:8643/healthz\": dial tcp 10.132.0.16:8643: connect: connection refused"
Sep 08 13:30:16.290232 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.290207    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:30:16.346471 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.346410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2015d03f-00cc-4908-97ba-c871c40fe283-proxy-tls\") pod \"2015d03f-00cc-4908-97ba-c871c40fe283\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") "
Sep 08 13:30:16.346471 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.346445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-kube-rbac-proxy-sar-config\") pod \"2015d03f-00cc-4908-97ba-c871c40fe283\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") "
Sep 08 13:30:16.346656 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.346483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-cabundle-cert\") pod \"2015d03f-00cc-4908-97ba-c871c40fe283\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") "
Sep 08 13:30:16.346656 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.346503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89nzr\" (UniqueName: \"kubernetes.io/projected/2015d03f-00cc-4908-97ba-c871c40fe283-kube-api-access-89nzr\") pod \"2015d03f-00cc-4908-97ba-c871c40fe283\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") "
Sep 08 13:30:16.346656 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.346528    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2015d03f-00cc-4908-97ba-c871c40fe283-kserve-provision-location\") pod \"2015d03f-00cc-4908-97ba-c871c40fe283\" (UID: \"2015d03f-00cc-4908-97ba-c871c40fe283\") "
Sep 08 13:30:16.346881 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.346804    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2015d03f-00cc-4908-97ba-c871c40fe283" (UID: "2015d03f-00cc-4908-97ba-c871c40fe283"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:16.346933 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.346873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2015d03f-00cc-4908-97ba-c871c40fe283" (UID: "2015d03f-00cc-4908-97ba-c871c40fe283"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:30:16.346972 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.346931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2015d03f-00cc-4908-97ba-c871c40fe283-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2015d03f-00cc-4908-97ba-c871c40fe283" (UID: "2015d03f-00cc-4908-97ba-c871c40fe283"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:30:16.348699 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.348661    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2015d03f-00cc-4908-97ba-c871c40fe283-kube-api-access-89nzr" (OuterVolumeSpecName: "kube-api-access-89nzr") pod "2015d03f-00cc-4908-97ba-c871c40fe283" (UID: "2015d03f-00cc-4908-97ba-c871c40fe283"). InnerVolumeSpecName "kube-api-access-89nzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:30:16.348788 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.348771    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2015d03f-00cc-4908-97ba-c871c40fe283-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2015d03f-00cc-4908-97ba-c871c40fe283" (UID: "2015d03f-00cc-4908-97ba-c871c40fe283"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:30:16.447210 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.447183    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:30:16.447210 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.447206    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2015d03f-00cc-4908-97ba-c871c40fe283-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:30:16.447537 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.447215    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89nzr\" (UniqueName: \"kubernetes.io/projected/2015d03f-00cc-4908-97ba-c871c40fe283-kube-api-access-89nzr\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:30:16.447537 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.447223    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2015d03f-00cc-4908-97ba-c871c40fe283-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:30:16.447537 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.447232    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2015d03f-00cc-4908-97ba-c871c40fe283-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:30:16.629071 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.628988    2581 generic.go:356] "Generic (PLEG): container finished" podID="2015d03f-00cc-4908-97ba-c871c40fe283" containerID="54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98" exitCode=0
Sep 08 13:30:16.629071 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.629041    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"
Sep 08 13:30:16.629224 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.629075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" event={"ID":"2015d03f-00cc-4908-97ba-c871c40fe283","Type":"ContainerDied","Data":"54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98"}
Sep 08 13:30:16.629224 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.629164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn" event={"ID":"2015d03f-00cc-4908-97ba-c871c40fe283","Type":"ContainerDied","Data":"b8abba28a452b78d78a7e32f9cfa0af7893f0ab01056c7df7042484803c45a84"}
Sep 08 13:30:16.629224 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.629193    2581 scope.go:117] "RemoveContainer" containerID="4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd"
Sep 08 13:30:16.638176 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.638161    2581 scope.go:117] "RemoveContainer" containerID="54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98"
Sep 08 13:30:16.645083 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.645067    2581 scope.go:117] "RemoveContainer" containerID="81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161"
Sep 08 13:30:16.649953 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.649925    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"]
Sep 08 13:30:16.652308 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.652292    2581 scope.go:117] "RemoveContainer" containerID="4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd"
Sep 08 13:30:16.652570 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:30:16.652544    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd\": container with ID starting with 4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd not found: ID does not exist" containerID="4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd"
Sep 08 13:30:16.652662 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.652578    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd"} err="failed to get container status \"4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd\": rpc error: code = NotFound desc = could not find container \"4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd\": container with ID starting with 4e8026be55b500cd6ee510d89f768137b1a0228f240e8d9712302ffc88f88fdd not found: ID does not exist"
Sep 08 13:30:16.652662 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.652599    2581 scope.go:117] "RemoveContainer" containerID="54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98"
Sep 08 13:30:16.652899 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:30:16.652879    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98\": container with ID starting with 54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98 not found: ID does not exist" containerID="54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98"
Sep 08 13:30:16.653008 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.652907    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98"} err="failed to get container status \"54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98\": rpc error: code = NotFound desc = could not find container \"54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98\": container with ID starting with 54d71335fb44256997737dc89873e9e0f15f6fa6df1061831e3f4f04a7e8ce98 not found: ID does not exist"
Sep 08 13:30:16.653008 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.652930    2581 scope.go:117] "RemoveContainer" containerID="81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161"
Sep 08 13:30:16.653312 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:30:16.653288    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161\": container with ID starting with 81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161 not found: ID does not exist" containerID="81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161"
Sep 08 13:30:16.653461 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.653320    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161"} err="failed to get container status \"81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161\": rpc error: code = NotFound desc = could not find container \"81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161\": container with ID starting with 81342ab9cccc033f9dfe716a6d2c88edd19c4e424887cf42999732c5ed078161 not found: ID does not exist"
Sep 08 13:30:16.654012 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.653994    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-5v7rn"]
Sep 08 13:30:16.821553 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:16.821516    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" path="/var/lib/kubelet/pods/2015d03f-00cc-4908-97ba-c871c40fe283/volumes"
Sep 08 13:30:17.945686 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945636    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"]
Sep 08 13:30:17.946131 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945885    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="storage-initializer"
Sep 08 13:30:17.946131 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945897    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="storage-initializer"
Sep 08 13:30:17.946131 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945909    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container"
Sep 08 13:30:17.946131 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945914    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container"
Sep 08 13:30:17.946131 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945919    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kube-rbac-proxy"
Sep 08 13:30:17.946131 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945925    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kube-rbac-proxy"
Sep 08 13:30:17.946131 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945973    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kserve-container"
Sep 08 13:30:17.946131 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.945980    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2015d03f-00cc-4908-97ba-c871c40fe283" containerName="kube-rbac-proxy"
Sep 08 13:30:17.950098 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.950081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:17.952556 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.952533    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:30:17.952692 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.952594    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:30:17.952692 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.952630    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:30:17.952818 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.952757    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 08 13:30:17.953218 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.953205    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:30:17.953279 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.953246    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 08 13:30:17.953354 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.953338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:30:17.959503 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:17.959483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"]
Sep 08 13:30:18.056940 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.056915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db2209b1-f638-4d11-a102-b8d363e4ecf7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.057048 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.056966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.057048 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.057032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2209b1-f638-4d11-a102-b8d363e4ecf7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.057159 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.057060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.057159 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.057104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jtp9\" (UniqueName: \"kubernetes.io/projected/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-api-access-9jtp9\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.157942 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.157918    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2209b1-f638-4d11-a102-b8d363e4ecf7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.158052 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.157957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.158052 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.157994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jtp9\" (UniqueName: \"kubernetes.io/projected/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-api-access-9jtp9\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.158052 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.158032    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db2209b1-f638-4d11-a102-b8d363e4ecf7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.158222 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.158088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.158310 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.158292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2209b1-f638-4d11-a102-b8d363e4ecf7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.158711 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.158691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.158783 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.158691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.160530 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.160512    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db2209b1-f638-4d11-a102-b8d363e4ecf7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.168920 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.168900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jtp9\" (UniqueName: \"kubernetes.io/projected/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-api-access-9jtp9\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-zx7hx\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.260745 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.260696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:18.536579 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.536495    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"]
Sep 08 13:30:18.540315 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:30:18.540288    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2209b1_f638_4d11_a102_b8d363e4ecf7.slice/crio-6bd2aae2a20132baa10bd1e23cb7e59fc787e3fa7dd0e685c4e4eca14b405818 WatchSource:0}: Error finding container 6bd2aae2a20132baa10bd1e23cb7e59fc787e3fa7dd0e685c4e4eca14b405818: Status 404 returned error can't find the container with id 6bd2aae2a20132baa10bd1e23cb7e59fc787e3fa7dd0e685c4e4eca14b405818
Sep 08 13:30:18.638288 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.638248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" event={"ID":"db2209b1-f638-4d11-a102-b8d363e4ecf7","Type":"ContainerStarted","Data":"bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd"}
Sep 08 13:30:18.638415 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:18.638303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" event={"ID":"db2209b1-f638-4d11-a102-b8d363e4ecf7","Type":"ContainerStarted","Data":"6bd2aae2a20132baa10bd1e23cb7e59fc787e3fa7dd0e685c4e4eca14b405818"}
Sep 08 13:30:22.652532 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:22.652440    2581 generic.go:356] "Generic (PLEG): container finished" podID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerID="bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd" exitCode=0
Sep 08 13:30:22.652532 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:22.652484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" event={"ID":"db2209b1-f638-4d11-a102-b8d363e4ecf7","Type":"ContainerDied","Data":"bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd"}
Sep 08 13:30:23.176475 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.176444    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"]
Sep 08 13:30:23.179892 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.179693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.181831 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.181781    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-3d858-predictor-serving-cert\""
Sep 08 13:30:23.182019 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.181992    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-3d858-kube-rbac-proxy-sar-config\""
Sep 08 13:30:23.199537 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.199511    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"]
Sep 08 13:30:23.294029 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.293991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e836f2b0-4433-4819-a615-dbea75b8570a-proxy-tls\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.294177 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.294041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.294177 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.294114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlgh8\" (UniqueName: \"kubernetes.io/projected/e836f2b0-4433-4819-a615-dbea75b8570a-kube-api-access-wlgh8\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.294253 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.294180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e836f2b0-4433-4819-a615-dbea75b8570a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.294253 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.294221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.394773 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.394739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e836f2b0-4433-4819-a615-dbea75b8570a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.394944 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.394786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.394944 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.394824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e836f2b0-4433-4819-a615-dbea75b8570a-proxy-tls\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.394944 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.394842    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.394944 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.394863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgh8\" (UniqueName: \"kubernetes.io/projected/e836f2b0-4433-4819-a615-dbea75b8570a-kube-api-access-wlgh8\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.395207 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.395184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e836f2b0-4433-4819-a615-dbea75b8570a-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.395496 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.395473    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-cabundle-cert\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.395572 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.395540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.397506 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.397482    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e836f2b0-4433-4819-a615-dbea75b8570a-proxy-tls\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.411136 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.411108    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlgh8\" (UniqueName: \"kubernetes.io/projected/e836f2b0-4433-4819-a615-dbea75b8570a-kube-api-access-wlgh8\") pod \"isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.494078 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.493991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:23.659245 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.659208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" event={"ID":"db2209b1-f638-4d11-a102-b8d363e4ecf7","Type":"ContainerStarted","Data":"172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9"}
Sep 08 13:30:23.659620 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.659271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" event={"ID":"db2209b1-f638-4d11-a102-b8d363e4ecf7","Type":"ContainerStarted","Data":"17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352"}
Sep 08 13:30:23.659620 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.659544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:23.659762 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.659688    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:23.660996 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.660969    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:30:23.683298 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.683250    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podStartSLOduration=2.941919982 podStartE2EDuration="6.683235675s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.741315693s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:23.681496807 +0000 UTC m=+853.385282656" watchObservedRunningTime="2026-09-08 13:30:23.683235675 +0000 UTC m=+853.387021525"
Sep 08 13:30:23.768913 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:23.768486    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"]
Sep 08 13:30:23.771117 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:30:23.771094    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode836f2b0_4433_4819_a615_dbea75b8570a.slice/crio-d946c6b7856015e863ff0a6e3b6df9a7d0de4e679eb479bda4cded9fec1d41c8 WatchSource:0}: Error finding container d946c6b7856015e863ff0a6e3b6df9a7d0de4e679eb479bda4cded9fec1d41c8: Status 404 returned error can't find the container with id d946c6b7856015e863ff0a6e3b6df9a7d0de4e679eb479bda4cded9fec1d41c8
Sep 08 13:30:24.664406 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:24.664364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" event={"ID":"e836f2b0-4433-4819-a615-dbea75b8570a","Type":"ContainerStarted","Data":"f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50"}
Sep 08 13:30:24.664833 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:24.664426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" event={"ID":"e836f2b0-4433-4819-a615-dbea75b8570a","Type":"ContainerStarted","Data":"d946c6b7856015e863ff0a6e3b6df9a7d0de4e679eb479bda4cded9fec1d41c8"}
Sep 08 13:30:24.664833 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:24.664626    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:30:27.676266 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:27.676181    2581 generic.go:356] "Generic (PLEG): container finished" podID="e836f2b0-4433-4819-a615-dbea75b8570a" containerID="f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50" exitCode=0
Sep 08 13:30:27.676266 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:27.676253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" event={"ID":"e836f2b0-4433-4819-a615-dbea75b8570a","Type":"ContainerDied","Data":"f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50"}
Sep 08 13:30:28.681320 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:28.681286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" event={"ID":"e836f2b0-4433-4819-a615-dbea75b8570a","Type":"ContainerStarted","Data":"6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b"}
Sep 08 13:30:28.681744 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:28.681350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" event={"ID":"e836f2b0-4433-4819-a615-dbea75b8570a","Type":"ContainerStarted","Data":"1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245"}
Sep 08 13:30:28.681744 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:28.681641    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:28.681828 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:28.681809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:28.683106 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:28.683082    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:30:28.703105 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:28.703059    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podStartSLOduration=2.161817734 podStartE2EDuration="5.703045163s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.541227429s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:30:28.701627211 +0000 UTC m=+858.405413079" watchObservedRunningTime="2026-09-08 13:30:28.703045163 +0000 UTC m=+858.406831023"
Sep 08 13:30:29.668989 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:29.668960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:30:29.669642 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:29.669613    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:30:29.684685 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:29.684648    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:30:34.690228 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:34.690196    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:30:34.690818 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:34.690791    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:30:39.669871 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:39.669832    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:30:44.691756 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:44.691706    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:30:49.669833 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:49.669791    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:30:54.690772 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:54.690727    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:30:59.670536 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:30:59.670491    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:31:04.690891 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:04.690798    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:31:09.670458 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:09.670417    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:31:14.691109 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:14.691068    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:31:19.670152 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:19.670113    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:31:24.691433 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:24.691392    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:31:29.670302 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:29.670261    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:31:34.691474 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:34.691436    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:31:39.669856 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:39.669820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:31:44.691364 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:44.691335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:31:47.946205 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:47.946172    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"]
Sep 08 13:31:47.946611 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:47.946432    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" containerID="cri-o://17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352" gracePeriod=30
Sep 08 13:31:47.946611 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:47.946473    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kube-rbac-proxy" containerID="cri-o://172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9" gracePeriod=30
Sep 08 13:31:48.979067 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:48.978961    2581 generic.go:356] "Generic (PLEG): container finished" podID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerID="172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9" exitCode=2
Sep 08 13:31:48.979067 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:48.978999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" event={"ID":"db2209b1-f638-4d11-a102-b8d363e4ecf7","Type":"ContainerDied","Data":"172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9"}
Sep 08 13:31:49.665585 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:49.665538    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
Sep 08 13:31:49.669758 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:49.669733    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
Sep 08 13:31:52.184347 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.184324    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:31:52.337111 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.337079    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-rbac-proxy-sar-config\") pod \"db2209b1-f638-4d11-a102-b8d363e4ecf7\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") "
Sep 08 13:31:52.337276 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.337136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2209b1-f638-4d11-a102-b8d363e4ecf7-kserve-provision-location\") pod \"db2209b1-f638-4d11-a102-b8d363e4ecf7\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") "
Sep 08 13:31:52.337276 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.337171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-cabundle-cert\") pod \"db2209b1-f638-4d11-a102-b8d363e4ecf7\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") "
Sep 08 13:31:52.337276 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.337189    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jtp9\" (UniqueName: \"kubernetes.io/projected/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-api-access-9jtp9\") pod \"db2209b1-f638-4d11-a102-b8d363e4ecf7\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") "
Sep 08 13:31:52.337276 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.337225    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db2209b1-f638-4d11-a102-b8d363e4ecf7-proxy-tls\") pod \"db2209b1-f638-4d11-a102-b8d363e4ecf7\" (UID: \"db2209b1-f638-4d11-a102-b8d363e4ecf7\") "
Sep 08 13:31:52.337533 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.337506    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db2209b1-f638-4d11-a102-b8d363e4ecf7" (UID: "db2209b1-f638-4d11-a102-b8d363e4ecf7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:52.337579 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.337529    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db2209b1-f638-4d11-a102-b8d363e4ecf7" (UID: "db2209b1-f638-4d11-a102-b8d363e4ecf7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:31:52.337579 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.337504    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db2209b1-f638-4d11-a102-b8d363e4ecf7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db2209b1-f638-4d11-a102-b8d363e4ecf7" (UID: "db2209b1-f638-4d11-a102-b8d363e4ecf7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:31:52.339483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.339459    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-api-access-9jtp9" (OuterVolumeSpecName: "kube-api-access-9jtp9") pod "db2209b1-f638-4d11-a102-b8d363e4ecf7" (UID: "db2209b1-f638-4d11-a102-b8d363e4ecf7"). InnerVolumeSpecName "kube-api-access-9jtp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:31:52.339571 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.339547    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db2209b1-f638-4d11-a102-b8d363e4ecf7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db2209b1-f638-4d11-a102-b8d363e4ecf7" (UID: "db2209b1-f638-4d11-a102-b8d363e4ecf7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:31:52.437890 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.437863    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db2209b1-f638-4d11-a102-b8d363e4ecf7-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.437890 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.437887    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.438037 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.437897    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jtp9\" (UniqueName: \"kubernetes.io/projected/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-api-access-9jtp9\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.438037 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.437906    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db2209b1-f638-4d11-a102-b8d363e4ecf7-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.438037 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.437915    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db2209b1-f638-4d11-a102-b8d363e4ecf7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:31:52.996569 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.996534    2581 generic.go:356] "Generic (PLEG): container finished" podID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerID="17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352" exitCode=0
Sep 08 13:31:52.996753 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.996589    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"
Sep 08 13:31:52.996753 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.996593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" event={"ID":"db2209b1-f638-4d11-a102-b8d363e4ecf7","Type":"ContainerDied","Data":"17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352"}
Sep 08 13:31:52.996753 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.996645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx" event={"ID":"db2209b1-f638-4d11-a102-b8d363e4ecf7","Type":"ContainerDied","Data":"6bd2aae2a20132baa10bd1e23cb7e59fc787e3fa7dd0e685c4e4eca14b405818"}
Sep 08 13:31:52.996753 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:52.996693    2581 scope.go:117] "RemoveContainer" containerID="172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9"
Sep 08 13:31:53.004305 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.004283    2581 scope.go:117] "RemoveContainer" containerID="17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352"
Sep 08 13:31:53.011423 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.011406    2581 scope.go:117] "RemoveContainer" containerID="bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd"
Sep 08 13:31:53.016948 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.016927    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"]
Sep 08 13:31:53.018987 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.018969    2581 scope.go:117] "RemoveContainer" containerID="172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9"
Sep 08 13:31:53.019265 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:31:53.019246    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9\": container with ID starting with 172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9 not found: ID does not exist" containerID="172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9"
Sep 08 13:31:53.019323 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.019272    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9"} err="failed to get container status \"172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9\": rpc error: code = NotFound desc = could not find container \"172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9\": container with ID starting with 172c4379e312a9704976b87d4cc7766340c7505f7715ea87b0d46495ba0792b9 not found: ID does not exist"
Sep 08 13:31:53.019323 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.019289    2581 scope.go:117] "RemoveContainer" containerID="17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352"
Sep 08 13:31:53.019572 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:31:53.019555    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352\": container with ID starting with 17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352 not found: ID does not exist" containerID="17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352"
Sep 08 13:31:53.019619 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.019578    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352"} err="failed to get container status \"17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352\": rpc error: code = NotFound desc = could not find container \"17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352\": container with ID starting with 17d06953ff89e633add13d7e5ddf85b308c8620bd8a1c50b5f0b3277e1c09352 not found: ID does not exist"
Sep 08 13:31:53.019619 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.019593    2581 scope.go:117] "RemoveContainer" containerID="bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd"
Sep 08 13:31:53.019847 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:31:53.019833    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd\": container with ID starting with bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd not found: ID does not exist" containerID="bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd"
Sep 08 13:31:53.019885 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.019852    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd"} err="failed to get container status \"bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd\": rpc error: code = NotFound desc = could not find container \"bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd\": container with ID starting with bfbec2faa63ee60a718fdacf4ae467a1869d5d6767c09d8d4744fb2cccf8cddd not found: ID does not exist"
Sep 08 13:31:53.024516 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.024494    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-zx7hx"]
Sep 08 13:31:53.175271 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175200    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"]
Sep 08 13:31:53.175457 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175446    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container"
Sep 08 13:31:53.175507 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175459    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container"
Sep 08 13:31:53.175507 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="storage-initializer"
Sep 08 13:31:53.175507 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175482    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="storage-initializer"
Sep 08 13:31:53.175507 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175490    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kube-rbac-proxy"
Sep 08 13:31:53.175507 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175496    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kube-rbac-proxy"
Sep 08 13:31:53.175717 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175535    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kserve-container"
Sep 08 13:31:53.175717 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.175543    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" containerName="kube-rbac-proxy"
Sep 08 13:31:53.179838 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.179806    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.182097 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.182075    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 08 13:31:53.182215 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.182102    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 08 13:31:53.191845 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.191824    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"]
Sep 08 13:31:53.244483 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.244450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.244656 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.244488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.244656 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.244526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.244656 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.244548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s9gf\" (UniqueName: \"kubernetes.io/projected/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-api-access-8s9gf\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.244656 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.244604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.345230 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.345179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.345388 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.345264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.345388 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.345288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.345388 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.345321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.345388 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.345347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9gf\" (UniqueName: \"kubernetes.io/projected/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-api-access-8s9gf\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.345612 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:31:53.345506    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 08 13:31:53.345612 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:31:53.345587    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls podName:4fa78f0b-ab43-4625-8cc5-2d1b988f2206 nodeName:}" failed. No retries permitted until 2026-09-08 13:31:53.845571237 +0000 UTC m=+943.549357064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls") pod "isvc-canary-stop-predictor-7b7754b5bf-ttgvj" (UID: "4fa78f0b-ab43-4625-8cc5-2d1b988f2206") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 08 13:31:53.345746 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.345717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.345996 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.345979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.346033 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.345983    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.353581 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.353562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9gf\" (UniqueName: \"kubernetes.io/projected/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-api-access-8s9gf\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.849312 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.849277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:53.851834 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:53.851806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-ttgvj\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:54.089817 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:54.089788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:31:54.365050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:54.364998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"]
Sep 08 13:31:54.367758 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:31:54.367731    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa78f0b_ab43_4625_8cc5_2d1b988f2206.slice/crio-053d083e2b879a0c335e40bb82ffcf3a9d3433173ae2bf17f0a39524682bedd8 WatchSource:0}: Error finding container 053d083e2b879a0c335e40bb82ffcf3a9d3433173ae2bf17f0a39524682bedd8: Status 404 returned error can't find the container with id 053d083e2b879a0c335e40bb82ffcf3a9d3433173ae2bf17f0a39524682bedd8
Sep 08 13:31:54.821451 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:54.821419    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db2209b1-f638-4d11-a102-b8d363e4ecf7" path="/var/lib/kubelet/pods/db2209b1-f638-4d11-a102-b8d363e4ecf7/volumes"
Sep 08 13:31:55.006608 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:55.006570    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" event={"ID":"4fa78f0b-ab43-4625-8cc5-2d1b988f2206","Type":"ContainerStarted","Data":"5c95281ad76b6d338735b6c73cfda22e302fe79f6051fafff07e618b0cfd9ef3"}
Sep 08 13:31:55.006785 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:55.006632    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" event={"ID":"4fa78f0b-ab43-4625-8cc5-2d1b988f2206","Type":"ContainerStarted","Data":"053d083e2b879a0c335e40bb82ffcf3a9d3433173ae2bf17f0a39524682bedd8"}
Sep 08 13:31:59.022215 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:59.022177    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerID="5c95281ad76b6d338735b6c73cfda22e302fe79f6051fafff07e618b0cfd9ef3" exitCode=0
Sep 08 13:31:59.022631 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:31:59.022249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" event={"ID":"4fa78f0b-ab43-4625-8cc5-2d1b988f2206","Type":"ContainerDied","Data":"5c95281ad76b6d338735b6c73cfda22e302fe79f6051fafff07e618b0cfd9ef3"}
Sep 08 13:32:00.028149 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:00.028114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" event={"ID":"4fa78f0b-ab43-4625-8cc5-2d1b988f2206","Type":"ContainerStarted","Data":"b5f401256d9a23be1c943a35428722c2086cfe1a23e8c540c24c4b2b8817e477"}
Sep 08 13:32:00.028516 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:00.028181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" event={"ID":"4fa78f0b-ab43-4625-8cc5-2d1b988f2206","Type":"ContainerStarted","Data":"429cdd60e286232f60fc825ab0957230dd152127927552661041f5835733cd7d"}
Sep 08 13:32:00.028516 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:00.028380    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:32:00.050815 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:00.050766    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podStartSLOduration=3.336101452 podStartE2EDuration="7.050751757s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.714650305s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:32:00.049535084 +0000 UTC m=+949.753320953" watchObservedRunningTime="2026-09-08 13:32:00.050751757 +0000 UTC m=+949.754537607"
Sep 08 13:32:01.036519 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:01.036486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:32:01.037585 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:01.037559    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:32:02.039759 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:02.039721    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:32:07.044367 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:07.044331    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:32:07.044816 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:07.044791    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:32:13.267438 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.267403    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"]
Sep 08 13:32:13.267911 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.267722    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" containerID="cri-o://1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245" gracePeriod=30
Sep 08 13:32:13.267911 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.267745    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kube-rbac-proxy" containerID="cri-o://6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b" gracePeriod=30
Sep 08 13:32:13.409558 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.409520    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"]
Sep 08 13:32:13.412584 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.412569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.415050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.415031    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-90436-kube-rbac-proxy-sar-config\""
Sep 08 13:32:13.415156 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.415052    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-90436-predictor-serving-cert\""
Sep 08 13:32:13.422951 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.422934    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"]
Sep 08 13:32:13.583375 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.583343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17a30d3-2828-4ded-bd3f-46c65d7cef01-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.583553 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.583383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zl8x\" (UniqueName: \"kubernetes.io/projected/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-api-access-9zl8x\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.583553 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.583421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.583553 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.583493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.583553 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.583530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.684431 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.684396    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17a30d3-2828-4ded-bd3f-46c65d7cef01-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.684431 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.684435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zl8x\" (UniqueName: \"kubernetes.io/projected/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-api-access-9zl8x\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.684634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.684467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.684634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.684487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.684634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.684600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.685024 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.684993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.685305 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.685278    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.685449 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.685429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.687164 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.687138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17a30d3-2828-4ded-bd3f-46c65d7cef01-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.693579 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.693539    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zl8x\" (UniqueName: \"kubernetes.io/projected/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-api-access-9zl8x\") pod \"isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.722411 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.722389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:13.995151 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:13.995066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"]
Sep 08 13:32:13.998648 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:32:13.998623    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17a30d3_2828_4ded_bd3f_46c65d7cef01.slice/crio-b7ac768e6ff2653c0c2ab68e8b4660957c952e247122f1686963b7a5dd2f4d1f WatchSource:0}: Error finding container b7ac768e6ff2653c0c2ab68e8b4660957c952e247122f1686963b7a5dd2f4d1f: Status 404 returned error can't find the container with id b7ac768e6ff2653c0c2ab68e8b4660957c952e247122f1686963b7a5dd2f4d1f
Sep 08 13:32:14.083291 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:14.083259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" event={"ID":"a17a30d3-2828-4ded-bd3f-46c65d7cef01","Type":"ContainerStarted","Data":"a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947"}
Sep 08 13:32:14.083389 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:14.083354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" event={"ID":"a17a30d3-2828-4ded-bd3f-46c65d7cef01","Type":"ContainerStarted","Data":"b7ac768e6ff2653c0c2ab68e8b4660957c952e247122f1686963b7a5dd2f4d1f"}
Sep 08 13:32:14.085760 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:14.085735    2581 generic.go:356] "Generic (PLEG): container finished" podID="e836f2b0-4433-4819-a615-dbea75b8570a" containerID="6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b" exitCode=2
Sep 08 13:32:14.085850 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:14.085789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" event={"ID":"e836f2b0-4433-4819-a615-dbea75b8570a","Type":"ContainerDied","Data":"6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b"}
Sep 08 13:32:14.685777 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:14.685734    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused"
Sep 08 13:32:14.691445 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:14.691412    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused"
Sep 08 13:32:17.045058 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:17.045016    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:32:17.945526 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:17.945469    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:32:18.103394 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.103363    2581 generic.go:356] "Generic (PLEG): container finished" podID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerID="a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947" exitCode=0
Sep 08 13:32:18.103866 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.103442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" event={"ID":"a17a30d3-2828-4ded-bd3f-46c65d7cef01","Type":"ContainerDied","Data":"a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947"}
Sep 08 13:32:18.105515 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.105493    2581 generic.go:356] "Generic (PLEG): container finished" podID="e836f2b0-4433-4819-a615-dbea75b8570a" containerID="1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245" exitCode=0
Sep 08 13:32:18.105624 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.105520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" event={"ID":"e836f2b0-4433-4819-a615-dbea75b8570a","Type":"ContainerDied","Data":"1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245"}
Sep 08 13:32:18.105624 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.105563    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"
Sep 08 13:32:18.105624 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.105580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj" event={"ID":"e836f2b0-4433-4819-a615-dbea75b8570a","Type":"ContainerDied","Data":"d946c6b7856015e863ff0a6e3b6df9a7d0de4e679eb479bda4cded9fec1d41c8"}
Sep 08 13:32:18.105624 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.105611    2581 scope.go:117] "RemoveContainer" containerID="6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b"
Sep 08 13:32:18.115559 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.115540    2581 scope.go:117] "RemoveContainer" containerID="1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245"
Sep 08 13:32:18.120689 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.120653    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e836f2b0-4433-4819-a615-dbea75b8570a-proxy-tls\") pod \"e836f2b0-4433-4819-a615-dbea75b8570a\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") "
Sep 08 13:32:18.120839 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.120818    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-kube-rbac-proxy-sar-config\") pod \"e836f2b0-4433-4819-a615-dbea75b8570a\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") "
Sep 08 13:32:18.120906 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.120869    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e836f2b0-4433-4819-a615-dbea75b8570a-kserve-provision-location\") pod \"e836f2b0-4433-4819-a615-dbea75b8570a\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") "
Sep 08 13:32:18.120959 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.120945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlgh8\" (UniqueName: \"kubernetes.io/projected/e836f2b0-4433-4819-a615-dbea75b8570a-kube-api-access-wlgh8\") pod \"e836f2b0-4433-4819-a615-dbea75b8570a\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") "
Sep 08 13:32:18.121012 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.120974    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-cabundle-cert\") pod \"e836f2b0-4433-4819-a615-dbea75b8570a\" (UID: \"e836f2b0-4433-4819-a615-dbea75b8570a\") "
Sep 08 13:32:18.121241 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.121217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e836f2b0-4433-4819-a615-dbea75b8570a" (UID: "e836f2b0-4433-4819-a615-dbea75b8570a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:32:18.121393 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.121364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e836f2b0-4433-4819-a615-dbea75b8570a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e836f2b0-4433-4819-a615-dbea75b8570a" (UID: "e836f2b0-4433-4819-a615-dbea75b8570a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:32:18.121452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.121417    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e836f2b0-4433-4819-a615-dbea75b8570a" (UID: "e836f2b0-4433-4819-a615-dbea75b8570a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:32:18.122764 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.122729    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e836f2b0-4433-4819-a615-dbea75b8570a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e836f2b0-4433-4819-a615-dbea75b8570a" (UID: "e836f2b0-4433-4819-a615-dbea75b8570a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:32:18.123232 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.123201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e836f2b0-4433-4819-a615-dbea75b8570a-kube-api-access-wlgh8" (OuterVolumeSpecName: "kube-api-access-wlgh8") pod "e836f2b0-4433-4819-a615-dbea75b8570a" (UID: "e836f2b0-4433-4819-a615-dbea75b8570a"). InnerVolumeSpecName "kube-api-access-wlgh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:32:18.133143 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.133126    2581 scope.go:117] "RemoveContainer" containerID="f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50"
Sep 08 13:32:18.146662 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.146647    2581 scope.go:117] "RemoveContainer" containerID="6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b"
Sep 08 13:32:18.147000 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:32:18.146971    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b\": container with ID starting with 6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b not found: ID does not exist" containerID="6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b"
Sep 08 13:32:18.147064 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.146997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b"} err="failed to get container status \"6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b\": rpc error: code = NotFound desc = could not find container \"6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b\": container with ID starting with 6566caa4ba72bc31c0140ee438fccc2c675e42df58b9824ffc8784ec0d85bb6b not found: ID does not exist"
Sep 08 13:32:18.147064 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.147013    2581 scope.go:117] "RemoveContainer" containerID="1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245"
Sep 08 13:32:18.147272 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:32:18.147251    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245\": container with ID starting with 1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245 not found: ID does not exist" containerID="1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245"
Sep 08 13:32:18.147316 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.147279    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245"} err="failed to get container status \"1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245\": rpc error: code = NotFound desc = could not find container \"1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245\": container with ID starting with 1f073c7f79d9ba264599c8e3ceced27aad43e8b131bcae770c3e53636241f245 not found: ID does not exist"
Sep 08 13:32:18.147316 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.147296    2581 scope.go:117] "RemoveContainer" containerID="f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50"
Sep 08 13:32:18.147568 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:32:18.147539    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50\": container with ID starting with f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50 not found: ID does not exist" containerID="f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50"
Sep 08 13:32:18.147662 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.147576    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50"} err="failed to get container status \"f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50\": rpc error: code = NotFound desc = could not find container \"f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50\": container with ID starting with f5d69fccfd7426e1f55850399361d5a7965f83df23cf55f02377f62baa1dfb50 not found: ID does not exist"
Sep 08 13:32:18.221853 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.221831    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e836f2b0-4433-4819-a615-dbea75b8570a-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:32:18.221971 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.221858    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:32:18.221971 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.221872    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e836f2b0-4433-4819-a615-dbea75b8570a-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:32:18.221971 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.221886    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlgh8\" (UniqueName: \"kubernetes.io/projected/e836f2b0-4433-4819-a615-dbea75b8570a-kube-api-access-wlgh8\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:32:18.221971 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.221899    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e836f2b0-4433-4819-a615-dbea75b8570a-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:32:18.427038 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.427010    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"]
Sep 08 13:32:18.430870 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.430849    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-3d858-predictor-554774999b-l8wkj"]
Sep 08 13:32:18.822001 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:18.821966    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" path="/var/lib/kubelet/pods/e836f2b0-4433-4819-a615-dbea75b8570a/volumes"
Sep 08 13:32:19.111355 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:19.111273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" event={"ID":"a17a30d3-2828-4ded-bd3f-46c65d7cef01","Type":"ContainerStarted","Data":"4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754"}
Sep 08 13:32:19.111355 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:19.111335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" event={"ID":"a17a30d3-2828-4ded-bd3f-46c65d7cef01","Type":"ContainerStarted","Data":"d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be"}
Sep 08 13:32:19.111827 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:19.111699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:19.111871 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:19.111847    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:19.113213 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:19.113183    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:32:19.131330 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:19.131286    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podStartSLOduration=2.314911949 podStartE2EDuration="6.131273345s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.816361396s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:32:19.129564145 +0000 UTC m=+968.833349991" watchObservedRunningTime="2026-09-08 13:32:19.131273345 +0000 UTC m=+968.835059193"
Sep 08 13:32:20.116628 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:20.116582    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:32:25.121059 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:25.121030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:32:25.121520 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:25.121492    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:32:27.045414 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:27.045378    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:32:35.121490 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:35.121404    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:32:37.045558 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:37.045511    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:32:45.121434 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:45.121395    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:32:47.045225 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:47.045181    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:32:55.121568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:55.121516    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:32:57.044925 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:32:57.044881    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:33:05.121573 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:05.121525    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:33:07.045025 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:07.044986    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused"
Sep 08 13:33:15.122300 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:15.122260    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:33:15.930133 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:15.930092    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"]
Sep 08 13:33:15.930396 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:15.930369    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container" containerID="cri-o://429cdd60e286232f60fc825ab0957230dd152127927552661041f5835733cd7d" gracePeriod=30
Sep 08 13:33:15.930490 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:15.930421    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kube-rbac-proxy" containerID="cri-o://b5f401256d9a23be1c943a35428722c2086cfe1a23e8c540c24c4b2b8817e477" gracePeriod=30
Sep 08 13:33:15.931175 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:15.930817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:33:16.318248 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:16.318214    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerID="b5f401256d9a23be1c943a35428722c2086cfe1a23e8c540c24c4b2b8817e477" exitCode=2
Sep 08 13:33:16.318618 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:16.318284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" event={"ID":"4fa78f0b-ab43-4625-8cc5-2d1b988f2206","Type":"ContainerDied","Data":"b5f401256d9a23be1c943a35428722c2086cfe1a23e8c540c24c4b2b8817e477"}
Sep 08 13:33:17.040243 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:17.040202    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused"
Sep 08 13:33:20.335221 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.335183    2581 generic.go:356] "Generic (PLEG): container finished" podID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerID="429cdd60e286232f60fc825ab0957230dd152127927552661041f5835733cd7d" exitCode=0
Sep 08 13:33:20.335549 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.335261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" event={"ID":"4fa78f0b-ab43-4625-8cc5-2d1b988f2206","Type":"ContainerDied","Data":"429cdd60e286232f60fc825ab0957230dd152127927552661041f5835733cd7d"}
Sep 08 13:33:20.372693 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.372657    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:33:20.473565 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.473482    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-rbac-proxy-sar-config\") pod \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") "
Sep 08 13:33:20.473565 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.473520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s9gf\" (UniqueName: \"kubernetes.io/projected/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-api-access-8s9gf\") pod \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") "
Sep 08 13:33:20.473565 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.473542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-cabundle-cert\") pod \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") "
Sep 08 13:33:20.473565 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.473565    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls\") pod \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") "
Sep 08 13:33:20.473907 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.473590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kserve-provision-location\") pod \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\" (UID: \"4fa78f0b-ab43-4625-8cc5-2d1b988f2206\") "
Sep 08 13:33:20.473956 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.473939    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fa78f0b-ab43-4625-8cc5-2d1b988f2206" (UID: "4fa78f0b-ab43-4625-8cc5-2d1b988f2206"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:33:20.473999 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.473952    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4fa78f0b-ab43-4625-8cc5-2d1b988f2206" (UID: "4fa78f0b-ab43-4625-8cc5-2d1b988f2206"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:20.473999 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.473950    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4fa78f0b-ab43-4625-8cc5-2d1b988f2206" (UID: "4fa78f0b-ab43-4625-8cc5-2d1b988f2206"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:33:20.475985 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.475962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-api-access-8s9gf" (OuterVolumeSpecName: "kube-api-access-8s9gf") pod "4fa78f0b-ab43-4625-8cc5-2d1b988f2206" (UID: "4fa78f0b-ab43-4625-8cc5-2d1b988f2206"). InnerVolumeSpecName "kube-api-access-8s9gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:33:20.475985 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.475969    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fa78f0b-ab43-4625-8cc5-2d1b988f2206" (UID: "4fa78f0b-ab43-4625-8cc5-2d1b988f2206"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:33:20.575043 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.575010    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s9gf\" (UniqueName: \"kubernetes.io/projected/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-api-access-8s9gf\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:33:20.575043 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.575038    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:33:20.575043 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.575047    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:33:20.575267 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.575058    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:33:20.575267 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:20.575066    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fa78f0b-ab43-4625-8cc5-2d1b988f2206-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:33:21.340237 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:21.340205    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"
Sep 08 13:33:21.340742 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:21.340203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj" event={"ID":"4fa78f0b-ab43-4625-8cc5-2d1b988f2206","Type":"ContainerDied","Data":"053d083e2b879a0c335e40bb82ffcf3a9d3433173ae2bf17f0a39524682bedd8"}
Sep 08 13:33:21.340742 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:21.340345    2581 scope.go:117] "RemoveContainer" containerID="b5f401256d9a23be1c943a35428722c2086cfe1a23e8c540c24c4b2b8817e477"
Sep 08 13:33:21.350538 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:21.350520    2581 scope.go:117] "RemoveContainer" containerID="429cdd60e286232f60fc825ab0957230dd152127927552661041f5835733cd7d"
Sep 08 13:33:21.357521 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:21.357490    2581 scope.go:117] "RemoveContainer" containerID="5c95281ad76b6d338735b6c73cfda22e302fe79f6051fafff07e618b0cfd9ef3"
Sep 08 13:33:21.359360 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:21.359311    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"]
Sep 08 13:33:21.365382 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:21.365366    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-ttgvj"]
Sep 08 13:33:22.821764 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:22.821728    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" path="/var/lib/kubelet/pods/4fa78f0b-ab43-4625-8cc5-2d1b988f2206/volumes"
Sep 08 13:33:25.122363 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:25.122321    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:33:33.407586 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407550    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"]
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407872    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kube-rbac-proxy"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407886    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kube-rbac-proxy"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407895    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407900    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407915    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407920    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407927    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="storage-initializer"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407932    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="storage-initializer"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407940    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="storage-initializer"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407945    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="storage-initializer"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407955    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kube-rbac-proxy"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407959    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kube-rbac-proxy"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.407994    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kube-rbac-proxy"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.408002    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kserve-container"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.408008    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fa78f0b-ab43-4625-8cc5-2d1b988f2206" containerName="kserve-container"
Sep 08 13:33:33.408039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.408014    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e836f2b0-4433-4819-a615-dbea75b8570a" containerName="kube-rbac-proxy"
Sep 08 13:33:33.412493 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.412472    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.414872 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.414849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d220f4-predictor-serving-cert\""
Sep 08 13:33:33.414968 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.414908    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-d220f4-kube-rbac-proxy-sar-config\""
Sep 08 13:33:33.420983 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.420960    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"]
Sep 08 13:33:33.464771 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.464737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kserve-provision-location\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.464771 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.464773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-cabundle-cert\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.464978 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.464794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.464978 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.464814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.464978 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.464830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2k2j\" (UniqueName: \"kubernetes.io/projected/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-api-access-h2k2j\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.566055 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.566010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-cabundle-cert\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.566055 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.566053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.566287 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:33:33.566154    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-d220f4-predictor-serving-cert: secret "isvc-primary-d220f4-predictor-serving-cert" not found
Sep 08 13:33:33.566287 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.566171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.566287 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.566205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2k2j\" (UniqueName: \"kubernetes.io/projected/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-api-access-h2k2j\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.566287 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:33:33.566227    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls podName:0c0df0d7-0f99-464d-814f-54b8a7c79fac nodeName:}" failed. No retries permitted until 2026-09-08 13:33:34.066205713 +0000 UTC m=+1043.769991557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls") pod "isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" (UID: "0c0df0d7-0f99-464d-814f-54b8a7c79fac") : secret "isvc-primary-d220f4-predictor-serving-cert" not found
Sep 08 13:33:33.566478 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.566314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kserve-provision-location\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.566632 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.566612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kserve-provision-location\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.566765 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.566746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-cabundle-cert\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.566821 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.566806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:33.575313 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:33.575292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2k2j\" (UniqueName: \"kubernetes.io/projected/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-api-access-h2k2j\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:34.071031 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:34.070998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:34.073452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:34.073432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls\") pod \"isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") " pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:34.322278 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:34.322163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:34.594962 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:34.594879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"]
Sep 08 13:33:34.601985 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:33:34.601958    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0df0d7_0f99_464d_814f_54b8a7c79fac.slice/crio-92aa6fce103cbccf38e17790d2ec7f9b49f3de1a55d55f3891b32ef38b9ee4af WatchSource:0}: Error finding container 92aa6fce103cbccf38e17790d2ec7f9b49f3de1a55d55f3891b32ef38b9ee4af: Status 404 returned error can't find the container with id 92aa6fce103cbccf38e17790d2ec7f9b49f3de1a55d55f3891b32ef38b9ee4af
Sep 08 13:33:35.122850 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:35.122822    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:33:35.398854 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:35.398758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" event={"ID":"0c0df0d7-0f99-464d-814f-54b8a7c79fac","Type":"ContainerStarted","Data":"78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5"}
Sep 08 13:33:35.399039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:35.398862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" event={"ID":"0c0df0d7-0f99-464d-814f-54b8a7c79fac","Type":"ContainerStarted","Data":"92aa6fce103cbccf38e17790d2ec7f9b49f3de1a55d55f3891b32ef38b9ee4af"}
Sep 08 13:33:38.411222 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:38.411182    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerID="78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5" exitCode=0
Sep 08 13:33:38.411569 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:38.411260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" event={"ID":"0c0df0d7-0f99-464d-814f-54b8a7c79fac","Type":"ContainerDied","Data":"78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5"}
Sep 08 13:33:39.416292 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:39.416257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" event={"ID":"0c0df0d7-0f99-464d-814f-54b8a7c79fac","Type":"ContainerStarted","Data":"67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e"}
Sep 08 13:33:39.416757 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:39.416316    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" event={"ID":"0c0df0d7-0f99-464d-814f-54b8a7c79fac","Type":"ContainerStarted","Data":"639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345"}
Sep 08 13:33:39.416757 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:39.416633    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:39.416897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:39.416768    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:39.418183 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:39.418157    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:33:39.436934 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:39.436888    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podStartSLOduration=2.894970723 podStartE2EDuration="6.436876093s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.54190537s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:33:39.435884674 +0000 UTC m=+1049.139670522" watchObservedRunningTime="2026-09-08 13:33:39.436876093 +0000 UTC m=+1049.140661941"
Sep 08 13:33:40.420138 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:40.420099    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:33:45.424786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:45.424749    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:33:45.425325 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:45.425297    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:33:55.425568 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:33:55.425522    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:34:03.729227 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:03.729146    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"]
Sep 08 13:34:03.729658 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:03.729373    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" containerID="cri-o://d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be" gracePeriod=30
Sep 08 13:34:03.729658 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:03.729388    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kube-rbac-proxy" containerID="cri-o://4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754" gracePeriod=30
Sep 08 13:34:04.507118 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:04.507084    2581 generic.go:356] "Generic (PLEG): container finished" podID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerID="4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754" exitCode=2
Sep 08 13:34:04.507342 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:04.507131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" event={"ID":"a17a30d3-2828-4ded-bd3f-46c65d7cef01","Type":"ContainerDied","Data":"4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754"}
Sep 08 13:34:05.117307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:05.117266    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused"
Sep 08 13:34:05.121537 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:05.121513    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused"
Sep 08 13:34:05.425435 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:05.425346    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:34:08.167199 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.167176    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:34:08.223396 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.223365    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17a30d3-2828-4ded-bd3f-46c65d7cef01-proxy-tls\") pod \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") "
Sep 08 13:34:08.223521 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.223413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zl8x\" (UniqueName: \"kubernetes.io/projected/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-api-access-9zl8x\") pod \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") "
Sep 08 13:34:08.223521 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.223449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kserve-provision-location\") pod \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") "
Sep 08 13:34:08.223521 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.223493    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-cabundle-cert\") pod \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") "
Sep 08 13:34:08.223700 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.223520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-rbac-proxy-sar-config\") pod \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\" (UID: \"a17a30d3-2828-4ded-bd3f-46c65d7cef01\") "
Sep 08 13:34:08.223829 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.223805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a17a30d3-2828-4ded-bd3f-46c65d7cef01" (UID: "a17a30d3-2828-4ded-bd3f-46c65d7cef01"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:34:08.223914 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.223852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a17a30d3-2828-4ded-bd3f-46c65d7cef01" (UID: "a17a30d3-2828-4ded-bd3f-46c65d7cef01"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:34:08.223914 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.223894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a17a30d3-2828-4ded-bd3f-46c65d7cef01" (UID: "a17a30d3-2828-4ded-bd3f-46c65d7cef01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:34:08.225517 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.225493    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17a30d3-2828-4ded-bd3f-46c65d7cef01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a17a30d3-2828-4ded-bd3f-46c65d7cef01" (UID: "a17a30d3-2828-4ded-bd3f-46c65d7cef01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:34:08.225593 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.225510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-api-access-9zl8x" (OuterVolumeSpecName: "kube-api-access-9zl8x") pod "a17a30d3-2828-4ded-bd3f-46c65d7cef01" (UID: "a17a30d3-2828-4ded-bd3f-46c65d7cef01"). InnerVolumeSpecName "kube-api-access-9zl8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:34:08.324280 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.324248    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:34:08.324280 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.324278    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:34:08.324280 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.324288    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a17a30d3-2828-4ded-bd3f-46c65d7cef01-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:34:08.324439 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.324296    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zl8x\" (UniqueName: \"kubernetes.io/projected/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kube-api-access-9zl8x\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:34:08.324439 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.324305    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a17a30d3-2828-4ded-bd3f-46c65d7cef01-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:34:08.522944 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.522911    2581 generic.go:356] "Generic (PLEG): container finished" podID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerID="d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be" exitCode=0
Sep 08 13:34:08.523085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.522974    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"
Sep 08 13:34:08.523085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.522997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" event={"ID":"a17a30d3-2828-4ded-bd3f-46c65d7cef01","Type":"ContainerDied","Data":"d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be"}
Sep 08 13:34:08.523085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.523072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr" event={"ID":"a17a30d3-2828-4ded-bd3f-46c65d7cef01","Type":"ContainerDied","Data":"b7ac768e6ff2653c0c2ab68e8b4660957c952e247122f1686963b7a5dd2f4d1f"}
Sep 08 13:34:08.523185 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.523101    2581 scope.go:117] "RemoveContainer" containerID="4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754"
Sep 08 13:34:08.532332 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.532312    2581 scope.go:117] "RemoveContainer" containerID="d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be"
Sep 08 13:34:08.539543 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.539528    2581 scope.go:117] "RemoveContainer" containerID="a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947"
Sep 08 13:34:08.546882 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.546863    2581 scope.go:117] "RemoveContainer" containerID="4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754"
Sep 08 13:34:08.547147 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:34:08.547129    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754\": container with ID starting with 4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754 not found: ID does not exist" containerID="4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754"
Sep 08 13:34:08.547213 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.547159    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754"} err="failed to get container status \"4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754\": rpc error: code = NotFound desc = could not find container \"4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754\": container with ID starting with 4e2464022329749319eab6443787762d3384cc93eee569306859ddba552d8754 not found: ID does not exist"
Sep 08 13:34:08.547213 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.547183    2581 scope.go:117] "RemoveContainer" containerID="d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be"
Sep 08 13:34:08.547436 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.547419    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"]
Sep 08 13:34:08.547495 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:34:08.547454    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be\": container with ID starting with d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be not found: ID does not exist" containerID="d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be"
Sep 08 13:34:08.547495 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.547481    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be"} err="failed to get container status \"d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be\": rpc error: code = NotFound desc = could not find container \"d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be\": container with ID starting with d41c903cb99787f9532fd2829a1ce5d5f698f5d6553b3cdeab18e3032a14d4be not found: ID does not exist"
Sep 08 13:34:08.547597 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.547500    2581 scope.go:117] "RemoveContainer" containerID="a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947"
Sep 08 13:34:08.547760 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:34:08.547743    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947\": container with ID starting with a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947 not found: ID does not exist" containerID="a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947"
Sep 08 13:34:08.547834 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.547764    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947"} err="failed to get container status \"a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947\": rpc error: code = NotFound desc = could not find container \"a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947\": container with ID starting with a9c2cc8f60e55cd3b52fd80c1cf2601ace6a399e16c83d8299755031397b1947 not found: ID does not exist"
Sep 08 13:34:08.554070 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.554048    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-90436-predictor-588584784-q82zr"]
Sep 08 13:34:08.821458 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:08.821431    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" path="/var/lib/kubelet/pods/a17a30d3-2828-4ded-bd3f-46c65d7cef01/volumes"
Sep 08 13:34:15.425301 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:15.425259    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:34:25.425580 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:25.425530    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:34:35.425940 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:35.425891    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:34:45.425240 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:45.425197    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:34:55.425820 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:34:55.425788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:35:18.612062 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612027    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"]
Sep 08 13:35:18.612486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612327    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="storage-initializer"
Sep 08 13:35:18.612486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612340    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="storage-initializer"
Sep 08 13:35:18.612486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612351    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kube-rbac-proxy"
Sep 08 13:35:18.612486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612357    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kube-rbac-proxy"
Sep 08 13:35:18.612486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612364    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container"
Sep 08 13:35:18.612486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612369    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container"
Sep 08 13:35:18.612486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612411    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kube-rbac-proxy"
Sep 08 13:35:18.612486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.612421    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a17a30d3-2828-4ded-bd3f-46c65d7cef01" containerName="kserve-container"
Sep 08 13:35:18.615390 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.615374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.618004 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.617974    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d220f4-predictor-serving-cert\""
Sep 08 13:35:18.618408 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.618389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-d220f4\""
Sep 08 13:35:18.618577 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.618393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-d220f4-dockercfg-qdfds\""
Sep 08 13:35:18.618650 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.618590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-d220f4-kube-rbac-proxy-sar-config\""
Sep 08 13:35:18.626356 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.626335    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"]
Sep 08 13:35:18.706406 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.706383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.706505 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.706408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kserve-provision-location\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.706505 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.706442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-cabundle-cert\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.706576 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.706498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.706576 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.706536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9s7b\" (UniqueName: \"kubernetes.io/projected/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-api-access-j9s7b\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.807510 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.807491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.807612 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.807522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9s7b\" (UniqueName: \"kubernetes.io/projected/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-api-access-j9s7b\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.807612 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.807549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.807734 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:18.807632    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-serving-cert: secret "isvc-secondary-d220f4-predictor-serving-cert" not found
Sep 08 13:35:18.807734 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.807657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kserve-provision-location\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.807734 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:18.807713    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls podName:7a69b3fc-2006-4c83-ba99-ca6dbb2c9387 nodeName:}" failed. No retries permitted until 2026-09-08 13:35:19.307694302 +0000 UTC m=+1149.011480143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls") pod "isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9" (UID: "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387") : secret "isvc-secondary-d220f4-predictor-serving-cert" not found
Sep 08 13:35:18.807890 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.807796    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-cabundle-cert\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.808076 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.808057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kserve-provision-location\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.808259 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.808243    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.808307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.808290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-cabundle-cert\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:18.816434 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:18.816407    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9s7b\" (UniqueName: \"kubernetes.io/projected/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-api-access-j9s7b\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:19.312233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:19.312195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:19.314708 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:19.314682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls\") pod \"isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") " pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:19.525705 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:19.525663    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:19.797077 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:19.797034    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"]
Sep 08 13:35:19.799730 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:35:19.799698    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a69b3fc_2006_4c83_ba99_ca6dbb2c9387.slice/crio-d083544cc10036a1b14429e646140e7351084be990178ab170172c07d45d36e5 WatchSource:0}: Error finding container d083544cc10036a1b14429e646140e7351084be990178ab170172c07d45d36e5: Status 404 returned error can't find the container with id d083544cc10036a1b14429e646140e7351084be990178ab170172c07d45d36e5
Sep 08 13:35:20.776899 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:20.776862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9" event={"ID":"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387","Type":"ContainerStarted","Data":"de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230"}
Sep 08 13:35:20.777049 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:20.776958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9" event={"ID":"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387","Type":"ContainerStarted","Data":"d083544cc10036a1b14429e646140e7351084be990178ab170172c07d45d36e5"}
Sep 08 13:35:23.789104 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:23.789082    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387/storage-initializer/0.log"
Sep 08 13:35:23.789455 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:23.789122    2581 generic.go:356] "Generic (PLEG): container finished" podID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" containerID="de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230" exitCode=1
Sep 08 13:35:23.789455 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:23.789203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9" event={"ID":"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387","Type":"ContainerDied","Data":"de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230"}
Sep 08 13:35:24.795127 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:24.795101    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387/storage-initializer/0.log"
Sep 08 13:35:24.795517 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:24.795164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9" event={"ID":"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387","Type":"ContainerStarted","Data":"dbeed17a96ebf557f04103db6b487e77875c49eb3a37601183944f8492437231"}
Sep 08 13:35:26.804237 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:26.804210    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387/storage-initializer/1.log"
Sep 08 13:35:26.804618 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:26.804603    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387/storage-initializer/0.log"
Sep 08 13:35:26.804663 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:26.804635    2581 generic.go:356] "Generic (PLEG): container finished" podID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" containerID="dbeed17a96ebf557f04103db6b487e77875c49eb3a37601183944f8492437231" exitCode=1
Sep 08 13:35:26.804733 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:26.804699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9" event={"ID":"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387","Type":"ContainerDied","Data":"dbeed17a96ebf557f04103db6b487e77875c49eb3a37601183944f8492437231"}
Sep 08 13:35:26.804780 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:26.804753    2581 scope.go:117] "RemoveContainer" containerID="de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230"
Sep 08 13:35:26.805082 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:26.805057    2581 scope.go:117] "RemoveContainer" containerID="de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230"
Sep 08 13:35:26.815772 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:26.815751    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_kserve-ci-e2e-test(7a69b3fc-2006-4c83-ba99-ca6dbb2c9387)\"" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9" podUID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387"
Sep 08 13:35:26.816001 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:26.815973    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_kserve-ci-e2e-test_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387_0 in pod sandbox d083544cc10036a1b14429e646140e7351084be990178ab170172c07d45d36e5 from index: no such id: 'de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230'" containerID="de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230"
Sep 08 13:35:26.816046 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:26.816013    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_kserve-ci-e2e-test_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387_0 in pod sandbox d083544cc10036a1b14429e646140e7351084be990178ab170172c07d45d36e5 from index: no such id: 'de22b4aa33a2d666542dd5c7bc2286afc2bb6ddc5b47857c7e21fcf6ce998230'"
Sep 08 13:35:27.809793 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:27.809766    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387/storage-initializer/1.log"
Sep 08 13:35:34.654586 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.654499    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"]
Sep 08 13:35:34.683222 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.683190    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"]
Sep 08 13:35:34.683429 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.683407    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" containerID="cri-o://639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345" gracePeriod=30
Sep 08 13:35:34.683517 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.683444    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kube-rbac-proxy" containerID="cri-o://67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e" gracePeriod=30
Sep 08 13:35:34.802128 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.802112    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387/storage-initializer/1.log"
Sep 08 13:35:34.804335 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.804318    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:34.837687 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.837641    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerID="67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e" exitCode=2
Sep 08 13:35:34.837808 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.837718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" event={"ID":"0c0df0d7-0f99-464d-814f-54b8a7c79fac","Type":"ContainerDied","Data":"67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e"}
Sep 08 13:35:34.838950 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.838935    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9_7a69b3fc-2006-4c83-ba99-ca6dbb2c9387/storage-initializer/1.log"
Sep 08 13:35:34.839016 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.839004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9" event={"ID":"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387","Type":"ContainerDied","Data":"d083544cc10036a1b14429e646140e7351084be990178ab170172c07d45d36e5"}
Sep 08 13:35:34.839066 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.839020    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"
Sep 08 13:35:34.839066 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.839045    2581 scope.go:117] "RemoveContainer" containerID="dbeed17a96ebf557f04103db6b487e77875c49eb3a37601183944f8492437231"
Sep 08 13:35:34.928342 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.928291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls\") pod \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") "
Sep 08 13:35:34.928342 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.928323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9s7b\" (UniqueName: \"kubernetes.io/projected/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-api-access-j9s7b\") pod \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") "
Sep 08 13:35:34.928482 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.928359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-cabundle-cert\") pod \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") "
Sep 08 13:35:34.928482 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.928417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-rbac-proxy-sar-config\") pod \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") "
Sep 08 13:35:34.928482 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.928446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kserve-provision-location\") pod \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\" (UID: \"7a69b3fc-2006-4c83-ba99-ca6dbb2c9387\") "
Sep 08 13:35:34.928791 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.928770    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" (UID: "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:34.928853 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.928780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" (UID: "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:34.928853 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.928789    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" (UID: "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:34.930360 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.930340    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" (UID: "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:34.930436 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:34.930383    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-api-access-j9s7b" (OuterVolumeSpecName: "kube-api-access-j9s7b") pod "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" (UID: "7a69b3fc-2006-4c83-ba99-ca6dbb2c9387"). InnerVolumeSpecName "kube-api-access-j9s7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:35.029411 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.029389    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:35.029411 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.029409    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9s7b\" (UniqueName: \"kubernetes.io/projected/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-api-access-j9s7b\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:35.029533 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.029421    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:35.029533 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.029429    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:35.029533 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.029438    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:35.179057 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.178983    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"]
Sep 08 13:35:35.184199 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.184168    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-d220f4-predictor-7c67b87d58-mnpz9"]
Sep 08 13:35:35.420600 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.420561    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.21:8643/healthz\": dial tcp 10.132.0.21:8643: connect: connection refused"
Sep 08 13:35:35.425868 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:35.425844    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused"
Sep 08 13:35:36.821276 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:36.821240    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" path="/var/lib/kubelet/pods/7a69b3fc-2006-4c83-ba99-ca6dbb2c9387/volumes"
Sep 08 13:35:38.729229 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.729199    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:35:38.852022 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.851963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kserve-provision-location\") pod \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") "
Sep 08 13:35:38.852145 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.852028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls\") pod \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") "
Sep 08 13:35:38.852145 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.852063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-rbac-proxy-sar-config\") pod \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") "
Sep 08 13:35:38.852145 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.852110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-cabundle-cert\") pod \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") "
Sep 08 13:35:38.852307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.852237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2k2j\" (UniqueName: \"kubernetes.io/projected/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-api-access-h2k2j\") pod \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\" (UID: \"0c0df0d7-0f99-464d-814f-54b8a7c79fac\") "
Sep 08 13:35:38.852307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.852245    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0c0df0d7-0f99-464d-814f-54b8a7c79fac" (UID: "0c0df0d7-0f99-464d-814f-54b8a7c79fac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:38.852464 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.852439    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:38.852464 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.852454    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0c0df0d7-0f99-464d-814f-54b8a7c79fac" (UID: "0c0df0d7-0f99-464d-814f-54b8a7c79fac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:38.852547 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.852518    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0c0df0d7-0f99-464d-814f-54b8a7c79fac" (UID: "0c0df0d7-0f99-464d-814f-54b8a7c79fac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:38.854452 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.854424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-api-access-h2k2j" (OuterVolumeSpecName: "kube-api-access-h2k2j") pod "0c0df0d7-0f99-464d-814f-54b8a7c79fac" (UID: "0c0df0d7-0f99-464d-814f-54b8a7c79fac"). InnerVolumeSpecName "kube-api-access-h2k2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:38.854723 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.854698    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0c0df0d7-0f99-464d-814f-54b8a7c79fac" (UID: "0c0df0d7-0f99-464d-814f-54b8a7c79fac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:38.856258 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.856236    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerID="639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345" exitCode=0
Sep 08 13:35:38.856347 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.856265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" event={"ID":"0c0df0d7-0f99-464d-814f-54b8a7c79fac","Type":"ContainerDied","Data":"639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345"}
Sep 08 13:35:38.856347 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.856304    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"
Sep 08 13:35:38.856347 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.856335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9" event={"ID":"0c0df0d7-0f99-464d-814f-54b8a7c79fac","Type":"ContainerDied","Data":"92aa6fce103cbccf38e17790d2ec7f9b49f3de1a55d55f3891b32ef38b9ee4af"}
Sep 08 13:35:38.856460 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.856374    2581 scope.go:117] "RemoveContainer" containerID="67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e"
Sep 08 13:35:38.866580 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.866563    2581 scope.go:117] "RemoveContainer" containerID="639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345"
Sep 08 13:35:38.873642 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.873625    2581 scope.go:117] "RemoveContainer" containerID="78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5"
Sep 08 13:35:38.877165 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.877144    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"]
Sep 08 13:35:38.880557 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.880539    2581 scope.go:117] "RemoveContainer" containerID="67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e"
Sep 08 13:35:38.880850 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:38.880833    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e\": container with ID starting with 67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e not found: ID does not exist" containerID="67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e"
Sep 08 13:35:38.880900 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.880857    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e"} err="failed to get container status \"67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e\": rpc error: code = NotFound desc = could not find container \"67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e\": container with ID starting with 67d869698dc190d5beaf7a81d8d4a6cf3b7afc1e58eb58dc4f7f808c7ebe550e not found: ID does not exist"
Sep 08 13:35:38.880900 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.880873    2581 scope.go:117] "RemoveContainer" containerID="639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345"
Sep 08 13:35:38.881134 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:38.881117    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345\": container with ID starting with 639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345 not found: ID does not exist" containerID="639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345"
Sep 08 13:35:38.881177 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.881140    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345"} err="failed to get container status \"639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345\": rpc error: code = NotFound desc = could not find container \"639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345\": container with ID starting with 639f300a29866e4e7cbe346b754edd30a29439d622f770444640b33c1716d345 not found: ID does not exist"
Sep 08 13:35:38.881177 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.881155    2581 scope.go:117] "RemoveContainer" containerID="78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5"
Sep 08 13:35:38.881379 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:38.881362    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5\": container with ID starting with 78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5 not found: ID does not exist" containerID="78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5"
Sep 08 13:35:38.881424 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.881382    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5"} err="failed to get container status \"78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5\": rpc error: code = NotFound desc = could not find container \"78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5\": container with ID starting with 78dfa047a708558306681c835c1c9b21ff222b0859e86a218ca816da4e4ae3a5 not found: ID does not exist"
Sep 08 13:35:38.883166 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.883146    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-d220f4-predictor-5f9f9f7fb5-jxlg9"]
Sep 08 13:35:38.953070 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.953045    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:38.953070 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.953067    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0c0df0d7-0f99-464d-814f-54b8a7c79fac-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:38.953252 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.953084    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2k2j\" (UniqueName: \"kubernetes.io/projected/0c0df0d7-0f99-464d-814f-54b8a7c79fac-kube-api-access-h2k2j\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:38.953252 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:38.953093    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0c0df0d7-0f99-464d-814f-54b8a7c79fac-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:39.852708 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"]
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852912    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852923    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852935    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" containerName="storage-initializer"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852941    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" containerName="storage-initializer"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852948    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" containerName="storage-initializer"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852954    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" containerName="storage-initializer"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852963    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kube-rbac-proxy"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852968    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kube-rbac-proxy"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852978    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="storage-initializer"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.852983    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="storage-initializer"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.853020    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kserve-container"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.853029    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" containerName="kube-rbac-proxy"
Sep 08 13:35:39.853085 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.853034    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" containerName="storage-initializer"
Sep 08 13:35:39.853446 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.853109    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a69b3fc-2006-4c83-ba99-ca6dbb2c9387" containerName="storage-initializer"
Sep 08 13:35:39.857326 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.857309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:39.859542 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.859520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-2bc5d9\""
Sep 08 13:35:39.859702 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.859570    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2bc5d9-kube-rbac-proxy-sar-config\""
Sep 08 13:35:39.859761 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.859726    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:35:39.859872 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.859832    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-2bc5d9-dockercfg-lms8d\""
Sep 08 13:35:39.859872 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.859847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-2bc5d9-predictor-serving-cert\""
Sep 08 13:35:39.860320 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.860302    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:35:39.860399 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.860324    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:35:39.866373 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.866352    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"]
Sep 08 13:35:39.960014 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.959981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbh7s\" (UniqueName: \"kubernetes.io/projected/c2101293-3822-4833-9720-5b6749c37989-kube-api-access-kbh7s\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:39.960125 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.960034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:39.960125 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.960092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2101293-3822-4833-9720-5b6749c37989-kserve-provision-location\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:39.960217 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.960133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-cabundle-cert\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:39.960217 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:39.960160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.060720 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.060694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-cabundle-cert\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.060825 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.060731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.060825 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.060761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbh7s\" (UniqueName: \"kubernetes.io/projected/c2101293-3822-4833-9720-5b6749c37989-kube-api-access-kbh7s\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.060825 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.060794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.060825 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.060814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2101293-3822-4833-9720-5b6749c37989-kserve-provision-location\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.061045 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:40.060978    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-serving-cert: secret "isvc-init-fail-2bc5d9-predictor-serving-cert" not found
Sep 08 13:35:40.061097 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:35:40.061050    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls podName:c2101293-3822-4833-9720-5b6749c37989 nodeName:}" failed. No retries permitted until 2026-09-08 13:35:40.561027727 +0000 UTC m=+1170.264813557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls") pod "isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc" (UID: "c2101293-3822-4833-9720-5b6749c37989") : secret "isvc-init-fail-2bc5d9-predictor-serving-cert" not found
Sep 08 13:35:40.061209 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.061185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2101293-3822-4833-9720-5b6749c37989-kserve-provision-location\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.061443 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.061419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-cabundle-cert\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.061607 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.061591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.069509 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.069491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbh7s\" (UniqueName: \"kubernetes.io/projected/c2101293-3822-4833-9720-5b6749c37989-kube-api-access-kbh7s\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.565033 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.564978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.567618 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.567581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls\") pod \"isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") " pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.768982 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.768947    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:40.823396 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:40.823365    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c0df0d7-0f99-464d-814f-54b8a7c79fac" path="/var/lib/kubelet/pods/0c0df0d7-0f99-464d-814f-54b8a7c79fac/volumes"
Sep 08 13:35:41.042946 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:41.042914    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"]
Sep 08 13:35:41.046046 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:35:41.046020    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2101293_3822_4833_9720_5b6749c37989.slice/crio-649024378b73f003d08b6c69af5bd6a4c3376394e18b37d0f93f245c3143c452 WatchSource:0}: Error finding container 649024378b73f003d08b6c69af5bd6a4c3376394e18b37d0f93f245c3143c452: Status 404 returned error can't find the container with id 649024378b73f003d08b6c69af5bd6a4c3376394e18b37d0f93f245c3143c452
Sep 08 13:35:41.871173 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:41.871133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc" event={"ID":"c2101293-3822-4833-9720-5b6749c37989","Type":"ContainerStarted","Data":"022cfe6f966b4d68300eca84306fca6760e6bc31020388f39b2e05b1acb4287a"}
Sep 08 13:35:41.871363 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:41.871201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc" event={"ID":"c2101293-3822-4833-9720-5b6749c37989","Type":"ContainerStarted","Data":"649024378b73f003d08b6c69af5bd6a4c3376394e18b37d0f93f245c3143c452"}
Sep 08 13:35:44.881657 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:44.881627    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc_c2101293-3822-4833-9720-5b6749c37989/storage-initializer/0.log"
Sep 08 13:35:44.882065 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:44.881691    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2101293-3822-4833-9720-5b6749c37989" containerID="022cfe6f966b4d68300eca84306fca6760e6bc31020388f39b2e05b1acb4287a" exitCode=1
Sep 08 13:35:44.882065 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:44.881762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc" event={"ID":"c2101293-3822-4833-9720-5b6749c37989","Type":"ContainerDied","Data":"022cfe6f966b4d68300eca84306fca6760e6bc31020388f39b2e05b1acb4287a"}
Sep 08 13:35:45.887128 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:45.887098    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc_c2101293-3822-4833-9720-5b6749c37989/storage-initializer/0.log"
Sep 08 13:35:45.887515 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:45.887194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc" event={"ID":"c2101293-3822-4833-9720-5b6749c37989","Type":"ContainerStarted","Data":"32a51237b752dc16d4f6190a65d6fb39320b50544d96d93ab4cdfe0d9c103567"}
Sep 08 13:35:49.884208 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:49.884178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"]
Sep 08 13:35:49.884764 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:49.884405    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc" podUID="c2101293-3822-4833-9720-5b6749c37989" containerName="storage-initializer" containerID="cri-o://32a51237b752dc16d4f6190a65d6fb39320b50544d96d93ab4cdfe0d9c103567" gracePeriod=30
Sep 08 13:35:50.906550 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:50.906523    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc_c2101293-3822-4833-9720-5b6749c37989/storage-initializer/1.log"
Sep 08 13:35:50.907026 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:50.906935    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc_c2101293-3822-4833-9720-5b6749c37989/storage-initializer/0.log"
Sep 08 13:35:50.907026 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:50.906965    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2101293-3822-4833-9720-5b6749c37989" containerID="32a51237b752dc16d4f6190a65d6fb39320b50544d96d93ab4cdfe0d9c103567" exitCode=1
Sep 08 13:35:50.907140 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:50.907027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc" event={"ID":"c2101293-3822-4833-9720-5b6749c37989","Type":"ContainerDied","Data":"32a51237b752dc16d4f6190a65d6fb39320b50544d96d93ab4cdfe0d9c103567"}
Sep 08 13:35:50.907140 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:50.907079    2581 scope.go:117] "RemoveContainer" containerID="022cfe6f966b4d68300eca84306fca6760e6bc31020388f39b2e05b1acb4287a"
Sep 08 13:35:50.927287 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:50.927270    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc_c2101293-3822-4833-9720-5b6749c37989/storage-initializer/1.log"
Sep 08 13:35:50.930386 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:50.930368    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:51.046830 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.046807    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-cabundle-cert\") pod \"c2101293-3822-4833-9720-5b6749c37989\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") "
Sep 08 13:35:51.046991 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.046843    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls\") pod \"c2101293-3822-4833-9720-5b6749c37989\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") "
Sep 08 13:35:51.046991 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.046877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbh7s\" (UniqueName: \"kubernetes.io/projected/c2101293-3822-4833-9720-5b6749c37989-kube-api-access-kbh7s\") pod \"c2101293-3822-4833-9720-5b6749c37989\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") "
Sep 08 13:35:51.046991 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.046897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2101293-3822-4833-9720-5b6749c37989-kserve-provision-location\") pod \"c2101293-3822-4833-9720-5b6749c37989\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") "
Sep 08 13:35:51.046991 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.046930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-kube-rbac-proxy-sar-config\") pod \"c2101293-3822-4833-9720-5b6749c37989\" (UID: \"c2101293-3822-4833-9720-5b6749c37989\") "
Sep 08 13:35:51.047253 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.047176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c2101293-3822-4833-9720-5b6749c37989" (UID: "c2101293-3822-4833-9720-5b6749c37989"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:51.047307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.047204    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2101293-3822-4833-9720-5b6749c37989-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c2101293-3822-4833-9720-5b6749c37989" (UID: "c2101293-3822-4833-9720-5b6749c37989"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:51.047307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.047296    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c2101293-3822-4833-9720-5b6749c37989" (UID: "c2101293-3822-4833-9720-5b6749c37989"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:51.049052 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.049019    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2101293-3822-4833-9720-5b6749c37989-kube-api-access-kbh7s" (OuterVolumeSpecName: "kube-api-access-kbh7s") pod "c2101293-3822-4833-9720-5b6749c37989" (UID: "c2101293-3822-4833-9720-5b6749c37989"). InnerVolumeSpecName "kube-api-access-kbh7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:51.049052 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.049019    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2101293-3822-4833-9720-5b6749c37989" (UID: "c2101293-3822-4833-9720-5b6749c37989"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:51.148165 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.148129    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2101293-3822-4833-9720-5b6749c37989-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:51.148165 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.148156    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbh7s\" (UniqueName: \"kubernetes.io/projected/c2101293-3822-4833-9720-5b6749c37989-kube-api-access-kbh7s\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:51.148165 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.148165    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2101293-3822-4833-9720-5b6749c37989-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:51.148369 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.148175    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:51.148369 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.148184    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2101293-3822-4833-9720-5b6749c37989-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:35:51.911510 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.911483    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc_c2101293-3822-4833-9720-5b6749c37989/storage-initializer/1.log"
Sep 08 13:35:51.911970 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.911588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc" event={"ID":"c2101293-3822-4833-9720-5b6749c37989","Type":"ContainerDied","Data":"649024378b73f003d08b6c69af5bd6a4c3376394e18b37d0f93f245c3143c452"}
Sep 08 13:35:51.911970 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.911638    2581 scope.go:117] "RemoveContainer" containerID="32a51237b752dc16d4f6190a65d6fb39320b50544d96d93ab4cdfe0d9c103567"
Sep 08 13:35:51.911970 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.911592    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"
Sep 08 13:35:51.951986 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.951953    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"]
Sep 08 13:35:51.955840 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:51.955807    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-2bc5d9-predictor-6c55fd645d-g2cpc"]
Sep 08 13:35:52.822031 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:52.821998    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2101293-3822-4833-9720-5b6749c37989" path="/var/lib/kubelet/pods/c2101293-3822-4833-9720-5b6749c37989/volumes"
Sep 08 13:35:55.112290 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.112257    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"]
Sep 08 13:35:55.112665 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.112492    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2101293-3822-4833-9720-5b6749c37989" containerName="storage-initializer"
Sep 08 13:35:55.112665 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.112502    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2101293-3822-4833-9720-5b6749c37989" containerName="storage-initializer"
Sep 08 13:35:55.112665 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.112515    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2101293-3822-4833-9720-5b6749c37989" containerName="storage-initializer"
Sep 08 13:35:55.112665 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.112520    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2101293-3822-4833-9720-5b6749c37989" containerName="storage-initializer"
Sep 08 13:35:55.112665 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.112561    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2101293-3822-4833-9720-5b6749c37989" containerName="storage-initializer"
Sep 08 13:35:55.112665 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.112569    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2101293-3822-4833-9720-5b6749c37989" containerName="storage-initializer"
Sep 08 13:35:55.117069 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.117054    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.119402 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.119380    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:35:55.119526 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.119483    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:35:55.119526 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.119511    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-9fd9b-kube-rbac-proxy-sar-config\""
Sep 08 13:35:55.119811 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.119515    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:35:55.119811 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.119540    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-9fd9b-predictor-serving-cert\""
Sep 08 13:35:55.120213 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.120197    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:35:55.120301 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.120220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:35:55.133436 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.133416    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"]
Sep 08 13:35:55.176564 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.176542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.176701 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.176570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-cabundle-cert\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.176701 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.176593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e094d8-64cb-466f-9ea0-1f370453ae82-proxy-tls\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.176701 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.176611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5e094d8-64cb-466f-9ea0-1f370453ae82-kserve-provision-location\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.176818 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.176712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8x9p\" (UniqueName: \"kubernetes.io/projected/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-api-access-d8x9p\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.277034 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.277009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.277130 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.277038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-cabundle-cert\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.277130 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.277071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e094d8-64cb-466f-9ea0-1f370453ae82-proxy-tls\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.277130 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.277100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5e094d8-64cb-466f-9ea0-1f370453ae82-kserve-provision-location\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.277253 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.277146    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8x9p\" (UniqueName: \"kubernetes.io/projected/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-api-access-d8x9p\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.277596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.277565    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5e094d8-64cb-466f-9ea0-1f370453ae82-kserve-provision-location\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.277753 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.277733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-cabundle-cert\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.277906 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.277887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.279679 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.279649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e094d8-64cb-466f-9ea0-1f370453ae82-proxy-tls\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.286423 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.286403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8x9p\" (UniqueName: \"kubernetes.io/projected/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-api-access-d8x9p\") pod \"raw-sklearn-9fd9b-predictor-84545f5548-44zmt\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") " pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.427274 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.427215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:35:55.706366 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.706283    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"]
Sep 08 13:35:55.709340 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:35:55.709308    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e094d8_64cb_466f_9ea0_1f370453ae82.slice/crio-ac4f8de3102acf3dd700b1b43daf6c5741d1683b1a8edc0430c52984f73b7318 WatchSource:0}: Error finding container ac4f8de3102acf3dd700b1b43daf6c5741d1683b1a8edc0430c52984f73b7318: Status 404 returned error can't find the container with id ac4f8de3102acf3dd700b1b43daf6c5741d1683b1a8edc0430c52984f73b7318
Sep 08 13:35:55.926933 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.926894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" event={"ID":"f5e094d8-64cb-466f-9ea0-1f370453ae82","Type":"ContainerStarted","Data":"8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491"}
Sep 08 13:35:55.927142 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:55.926955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" event={"ID":"f5e094d8-64cb-466f-9ea0-1f370453ae82","Type":"ContainerStarted","Data":"ac4f8de3102acf3dd700b1b43daf6c5741d1683b1a8edc0430c52984f73b7318"}
Sep 08 13:35:59.058731 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.058699    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"]
Sep 08 13:35:59.062060 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.062034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.064276 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.064251    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-73c96-predictor-serving-cert\""
Sep 08 13:35:59.064392 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.064372    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-73c96-kube-rbac-proxy-sar-config\""
Sep 08 13:35:59.075017 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.074994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"]
Sep 08 13:35:59.103267 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.103234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c215706-e61e-4f71-938f-da96ddc3e79b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.103389 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.103275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.103389 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.103303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.103510 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.103411    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c215706-e61e-4f71-938f-da96ddc3e79b-proxy-tls\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.103510 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.103468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvf76\" (UniqueName: \"kubernetes.io/projected/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-api-access-xvf76\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.204053 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.204026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvf76\" (UniqueName: \"kubernetes.io/projected/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-api-access-xvf76\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.204158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.204061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c215706-e61e-4f71-938f-da96ddc3e79b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.204158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.204088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.204158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.204111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.204293 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.204168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c215706-e61e-4f71-938f-da96ddc3e79b-proxy-tls\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.204494 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.204475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c215706-e61e-4f71-938f-da96ddc3e79b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.204806 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.204783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-cabundle-cert\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.204865 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.204785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.206661 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.206639    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c215706-e61e-4f71-938f-da96ddc3e79b-proxy-tls\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.211687 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.211643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvf76\" (UniqueName: \"kubernetes.io/projected/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-api-access-xvf76\") pod \"isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.373907 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.373814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:35:59.654262 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.654186    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"]
Sep 08 13:35:59.656911 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:35:59.656883    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c215706_e61e_4f71_938f_da96ddc3e79b.slice/crio-58c4ceba9753ea4895374212659ac730ec80b80c6abed20964d231737e84721c WatchSource:0}: Error finding container 58c4ceba9753ea4895374212659ac730ec80b80c6abed20964d231737e84721c: Status 404 returned error can't find the container with id 58c4ceba9753ea4895374212659ac730ec80b80c6abed20964d231737e84721c
Sep 08 13:35:59.943146 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.943070    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerID="8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491" exitCode=0
Sep 08 13:35:59.943303 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.943154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" event={"ID":"f5e094d8-64cb-466f-9ea0-1f370453ae82","Type":"ContainerDied","Data":"8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491"}
Sep 08 13:35:59.944777 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.944753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" event={"ID":"2c215706-e61e-4f71-938f-da96ddc3e79b","Type":"ContainerStarted","Data":"f0cf06923a435aa3ccd3e08c0034e9502ddbc6659434a90e70eb2bd1616d83a3"}
Sep 08 13:35:59.944905 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:35:59.944806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" event={"ID":"2c215706-e61e-4f71-938f-da96ddc3e79b","Type":"ContainerStarted","Data":"58c4ceba9753ea4895374212659ac730ec80b80c6abed20964d231737e84721c"}
Sep 08 13:36:00.951018 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:00.950987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" event={"ID":"f5e094d8-64cb-466f-9ea0-1f370453ae82","Type":"ContainerStarted","Data":"7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b"}
Sep 08 13:36:00.951405 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:00.951049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" event={"ID":"f5e094d8-64cb-466f-9ea0-1f370453ae82","Type":"ContainerStarted","Data":"13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99"}
Sep 08 13:36:00.951405 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:00.951320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:36:00.951519 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:00.951448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:36:00.952724 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:00.952654    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:36:00.970761 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:00.970697    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podStartSLOduration=2.323319021 podStartE2EDuration="5.970662327s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.647343306s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:00.970349444 +0000 UTC m=+1190.674135294" watchObservedRunningTime="2026-09-08 13:36:00.970662327 +0000 UTC m=+1190.674448177"
Sep 08 13:36:01.954437 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:01.954397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:36:02.962762 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:02.962724    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:36:03.967599 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:03.967563    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerID="f0cf06923a435aa3ccd3e08c0034e9502ddbc6659434a90e70eb2bd1616d83a3" exitCode=0
Sep 08 13:36:03.967982 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:03.967604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" event={"ID":"2c215706-e61e-4f71-938f-da96ddc3e79b","Type":"ContainerDied","Data":"f0cf06923a435aa3ccd3e08c0034e9502ddbc6659434a90e70eb2bd1616d83a3"}
Sep 08 13:36:04.973401 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:04.973370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" event={"ID":"2c215706-e61e-4f71-938f-da96ddc3e79b","Type":"ContainerStarted","Data":"c70d9b5f05be85324430908f75dac8b594ce26d1b2d44696d7eee3feacdba822"}
Sep 08 13:36:04.973848 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:04.973428    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" event={"ID":"2c215706-e61e-4f71-938f-da96ddc3e79b","Type":"ContainerStarted","Data":"4e6b281a35ba77387e3306f602d3b7aaaf9e7b2ea224d3d358b03549c2e13b40"}
Sep 08 13:36:04.973848 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:04.973632    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:36:04.993565 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:04.993511    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podStartSLOduration=2.3705182320000002 podStartE2EDuration="5.99349648s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.622978248s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:04.991955507 +0000 UTC m=+1194.695741374" watchObservedRunningTime="2026-09-08 13:36:04.99349648 +0000 UTC m=+1194.697282327"
Sep 08 13:36:05.976916 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:05.976880    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:36:05.978064 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:05.978037    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:36:06.980262 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:06.980225    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:36:07.967197 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:07.967172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:36:07.967646 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:07.967621    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:36:11.986341 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:11.986314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:36:11.987016 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:11.986987    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:36:17.968162 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:17.968106    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:36:21.987050 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:21.987004    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:36:27.967622 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:27.967583    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:36:31.987258 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:31.987215    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:36:37.968163 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:37.968117    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:36:41.987182 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:41.987142    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:36:47.968587 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:47.968548    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:36:51.987737 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:51.987689    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:36:57.968559 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:36:57.968523    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:37:01.987459 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:01.987362    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:37:07.967842 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:07.967801    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:37:11.987072 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:11.987030    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:37:17.968886 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:17.968847    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:37:21.987839 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:21.987796    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:37:25.209479 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:25.209444    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"]
Sep 08 13:37:25.209885 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:25.209717    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" containerID="cri-o://13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99" gracePeriod=30
Sep 08 13:37:25.209885 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:25.209759    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kube-rbac-proxy" containerID="cri-o://7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b" gracePeriod=30
Sep 08 13:37:26.263278 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:26.263242    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerID="7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b" exitCode=2
Sep 08 13:37:26.263645 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:26.263313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" event={"ID":"f5e094d8-64cb-466f-9ea0-1f370453ae82","Type":"ContainerDied","Data":"7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b"}
Sep 08 13:37:26.817438 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:26.817391    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:37:27.963615 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:27.963575    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 08 13:37:27.967931 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:27.967906    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 08 13:37:29.549157 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.549133    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:37:29.662941 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.662877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8x9p\" (UniqueName: \"kubernetes.io/projected/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-api-access-d8x9p\") pod \"f5e094d8-64cb-466f-9ea0-1f370453ae82\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") "
Sep 08 13:37:29.662941 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.662930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e094d8-64cb-466f-9ea0-1f370453ae82-proxy-tls\") pod \"f5e094d8-64cb-466f-9ea0-1f370453ae82\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") "
Sep 08 13:37:29.663095 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.662966    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5e094d8-64cb-466f-9ea0-1f370453ae82-kserve-provision-location\") pod \"f5e094d8-64cb-466f-9ea0-1f370453ae82\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") "
Sep 08 13:37:29.663095 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.662992    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-rbac-proxy-sar-config\") pod \"f5e094d8-64cb-466f-9ea0-1f370453ae82\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") "
Sep 08 13:37:29.663095 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.663035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-cabundle-cert\") pod \"f5e094d8-64cb-466f-9ea0-1f370453ae82\" (UID: \"f5e094d8-64cb-466f-9ea0-1f370453ae82\") "
Sep 08 13:37:29.663359 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.663321    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5e094d8-64cb-466f-9ea0-1f370453ae82" (UID: "f5e094d8-64cb-466f-9ea0-1f370453ae82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:37:29.663359 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.663347    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5e094d8-64cb-466f-9ea0-1f370453ae82-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5e094d8-64cb-466f-9ea0-1f370453ae82" (UID: "f5e094d8-64cb-466f-9ea0-1f370453ae82"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:37:29.663552 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.663437    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f5e094d8-64cb-466f-9ea0-1f370453ae82" (UID: "f5e094d8-64cb-466f-9ea0-1f370453ae82"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:37:29.665062 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.665038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e094d8-64cb-466f-9ea0-1f370453ae82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5e094d8-64cb-466f-9ea0-1f370453ae82" (UID: "f5e094d8-64cb-466f-9ea0-1f370453ae82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:37:29.665158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.665140    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-api-access-d8x9p" (OuterVolumeSpecName: "kube-api-access-d8x9p") pod "f5e094d8-64cb-466f-9ea0-1f370453ae82" (UID: "f5e094d8-64cb-466f-9ea0-1f370453ae82"). InnerVolumeSpecName "kube-api-access-d8x9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:37:29.763852 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.763822    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.763852 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.763847    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8x9p\" (UniqueName: \"kubernetes.io/projected/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-api-access-d8x9p\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.763852 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.763856    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5e094d8-64cb-466f-9ea0-1f370453ae82-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.764068 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.763867    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5e094d8-64cb-466f-9ea0-1f370453ae82-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.764068 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:29.763875    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5e094d8-64cb-466f-9ea0-1f370453ae82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:37:30.279294 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.279260    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerID="13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99" exitCode=0
Sep 08 13:37:30.279432 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.279322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" event={"ID":"f5e094d8-64cb-466f-9ea0-1f370453ae82","Type":"ContainerDied","Data":"13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99"}
Sep 08 13:37:30.279432 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.279374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt" event={"ID":"f5e094d8-64cb-466f-9ea0-1f370453ae82","Type":"ContainerDied","Data":"ac4f8de3102acf3dd700b1b43daf6c5741d1683b1a8edc0430c52984f73b7318"}
Sep 08 13:37:30.279432 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.279328    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"
Sep 08 13:37:30.279432 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.279408    2581 scope.go:117] "RemoveContainer" containerID="7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b"
Sep 08 13:37:30.287791 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.287766    2581 scope.go:117] "RemoveContainer" containerID="13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99"
Sep 08 13:37:30.295558 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.295540    2581 scope.go:117] "RemoveContainer" containerID="8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491"
Sep 08 13:37:30.304078 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.304045    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"]
Sep 08 13:37:30.307440 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.307418    2581 scope.go:117] "RemoveContainer" containerID="7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b"
Sep 08 13:37:30.307825 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:37:30.307789    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b\": container with ID starting with 7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b not found: ID does not exist" containerID="7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b"
Sep 08 13:37:30.307921 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.307839    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b"} err="failed to get container status \"7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b\": rpc error: code = NotFound desc = could not find container \"7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b\": container with ID starting with 7a6796579aaa92e9799572b665e6022ba30dd34688d1b32546cf5049e21c285b not found: ID does not exist"
Sep 08 13:37:30.307921 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.307864    2581 scope.go:117] "RemoveContainer" containerID="13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99"
Sep 08 13:37:30.308220 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:37:30.308196    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99\": container with ID starting with 13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99 not found: ID does not exist" containerID="13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99"
Sep 08 13:37:30.308280 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.308227    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99"} err="failed to get container status \"13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99\": rpc error: code = NotFound desc = could not find container \"13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99\": container with ID starting with 13e2e296c22754590fd5eb40be90179ea0e0014639800486c5c4958c35488a99 not found: ID does not exist"
Sep 08 13:37:30.308280 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.308243    2581 scope.go:117] "RemoveContainer" containerID="8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491"
Sep 08 13:37:30.308502 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:37:30.308486    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491\": container with ID starting with 8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491 not found: ID does not exist" containerID="8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491"
Sep 08 13:37:30.308566 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.308504    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491"} err="failed to get container status \"8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491\": rpc error: code = NotFound desc = could not find container \"8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491\": container with ID starting with 8421a5f712d2624b7e2dd1f22531ff7392b3a54623867f71da8a195b8877b491 not found: ID does not exist"
Sep 08 13:37:30.308927 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.308908    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-9fd9b-predictor-84545f5548-44zmt"]
Sep 08 13:37:30.821755 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:30.821715    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" path="/var/lib/kubelet/pods/f5e094d8-64cb-466f-9ea0-1f370453ae82/volumes"
Sep 08 13:37:36.817838 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:36.817797    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:37:46.817517 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:46.817470    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:37:56.817497 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:37:56.817445    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:38:06.817627 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:06.817579    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:38:16.818295 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:16.818245    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:38:26.817605 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:26.817563    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:38:36.821912 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:36.821831    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:38:39.142436 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:39.142407    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"]
Sep 08 13:38:39.142924 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:39.142607    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" containerID="cri-o://4e6b281a35ba77387e3306f602d3b7aaaf9e7b2ea224d3d358b03549c2e13b40" gracePeriod=30
Sep 08 13:38:39.142924 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:39.142682    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kube-rbac-proxy" containerID="cri-o://c70d9b5f05be85324430908f75dac8b594ce26d1b2d44696d7eee3feacdba822" gracePeriod=30
Sep 08 13:38:39.516721 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:39.516630    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerID="c70d9b5f05be85324430908f75dac8b594ce26d1b2d44696d7eee3feacdba822" exitCode=2
Sep 08 13:38:39.516839 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:39.516709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" event={"ID":"2c215706-e61e-4f71-938f-da96ddc3e79b","Type":"ContainerDied","Data":"c70d9b5f05be85324430908f75dac8b594ce26d1b2d44696d7eee3feacdba822"}
Sep 08 13:38:41.980865 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:41.980829    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 08 13:38:44.378729 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.378693    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"]
Sep 08 13:38:44.379101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.378978    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container"
Sep 08 13:38:44.379101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.378988    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container"
Sep 08 13:38:44.379101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.379001    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="storage-initializer"
Sep 08 13:38:44.379101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.379007    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="storage-initializer"
Sep 08 13:38:44.379101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.379022    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kube-rbac-proxy"
Sep 08 13:38:44.379101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.379028    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kube-rbac-proxy"
Sep 08 13:38:44.379101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.379086    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kube-rbac-proxy"
Sep 08 13:38:44.379101 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.379095    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e094d8-64cb-466f-9ea0-1f370453ae82" containerName="kserve-container"
Sep 08 13:38:44.382017 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.381980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.386467 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.386443    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 08 13:38:44.386739 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.386719    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 08 13:38:44.400334 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.400311    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"]
Sep 08 13:38:44.457604 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.457553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea0edf9-8272-4066-9693-ccbe841193f1-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.457740 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.457617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea0edf9-8272-4066-9693-ccbe841193f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.457740 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.457700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.457814 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.457745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.457814 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.457766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd7x7\" (UniqueName: \"kubernetes.io/projected/eea0edf9-8272-4066-9693-ccbe841193f1-kube-api-access-sd7x7\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.558469 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.558435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea0edf9-8272-4066-9693-ccbe841193f1-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.558647 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.558485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea0edf9-8272-4066-9693-ccbe841193f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.558647 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.558524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.558647 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.558564    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.558647 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.558589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd7x7\" (UniqueName: \"kubernetes.io/projected/eea0edf9-8272-4066-9693-ccbe841193f1-kube-api-access-sd7x7\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.558938 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.558916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea0edf9-8272-4066-9693-ccbe841193f1-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.559277 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.559255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.559378 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.559264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.561195 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.561173    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea0edf9-8272-4066-9693-ccbe841193f1-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.581847 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.581818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd7x7\" (UniqueName: \"kubernetes.io/projected/eea0edf9-8272-4066-9693-ccbe841193f1-kube-api-access-sd7x7\") pod \"isvc-canary-create-predictor-8488566dd5-t6j5l\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.691702 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.691611    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:44.964605 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:44.964531    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"]
Sep 08 13:38:44.967434 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:38:44.967405    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea0edf9_8272_4066_9693_ccbe841193f1.slice/crio-1775673e2b908bcf76448868a03b146f100f035282700462b06552f7477a9b75 WatchSource:0}: Error finding container 1775673e2b908bcf76448868a03b146f100f035282700462b06552f7477a9b75: Status 404 returned error can't find the container with id 1775673e2b908bcf76448868a03b146f100f035282700462b06552f7477a9b75
Sep 08 13:38:45.539297 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:45.539262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" event={"ID":"eea0edf9-8272-4066-9693-ccbe841193f1","Type":"ContainerStarted","Data":"ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66"}
Sep 08 13:38:45.539778 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:45.539328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" event={"ID":"eea0edf9-8272-4066-9693-ccbe841193f1","Type":"ContainerStarted","Data":"1775673e2b908bcf76448868a03b146f100f035282700462b06552f7477a9b75"}
Sep 08 13:38:46.818308 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:46.818262    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 08 13:38:46.981054 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:46.981018    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Sep 08 13:38:48.552437 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.552401    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerID="4e6b281a35ba77387e3306f602d3b7aaaf9e7b2ea224d3d358b03549c2e13b40" exitCode=0
Sep 08 13:38:48.552790 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.552440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" event={"ID":"2c215706-e61e-4f71-938f-da96ddc3e79b","Type":"ContainerDied","Data":"4e6b281a35ba77387e3306f602d3b7aaaf9e7b2ea224d3d358b03549c2e13b40"}
Sep 08 13:38:48.584360 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.584335    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:38:48.688723 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.688600    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c215706-e61e-4f71-938f-da96ddc3e79b-proxy-tls\") pod \"2c215706-e61e-4f71-938f-da96ddc3e79b\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") "
Sep 08 13:38:48.688723 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.688646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c215706-e61e-4f71-938f-da96ddc3e79b-kserve-provision-location\") pod \"2c215706-e61e-4f71-938f-da96ddc3e79b\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") "
Sep 08 13:38:48.688723 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.688664    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-rbac-proxy-sar-config\") pod \"2c215706-e61e-4f71-938f-da96ddc3e79b\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") "
Sep 08 13:38:48.688995 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.688728    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvf76\" (UniqueName: \"kubernetes.io/projected/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-api-access-xvf76\") pod \"2c215706-e61e-4f71-938f-da96ddc3e79b\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") "
Sep 08 13:38:48.688995 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.688759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-cabundle-cert\") pod \"2c215706-e61e-4f71-938f-da96ddc3e79b\" (UID: \"2c215706-e61e-4f71-938f-da96ddc3e79b\") "
Sep 08 13:38:48.689105 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.689007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c215706-e61e-4f71-938f-da96ddc3e79b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c215706-e61e-4f71-938f-da96ddc3e79b" (UID: "2c215706-e61e-4f71-938f-da96ddc3e79b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:38:48.689105 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.689076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c215706-e61e-4f71-938f-da96ddc3e79b" (UID: "2c215706-e61e-4f71-938f-da96ddc3e79b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:38:48.689105 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.689095    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c215706-e61e-4f71-938f-da96ddc3e79b" (UID: "2c215706-e61e-4f71-938f-da96ddc3e79b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:38:48.690943 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.690926    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c215706-e61e-4f71-938f-da96ddc3e79b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c215706-e61e-4f71-938f-da96ddc3e79b" (UID: "2c215706-e61e-4f71-938f-da96ddc3e79b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:38:48.691067 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.691046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-api-access-xvf76" (OuterVolumeSpecName: "kube-api-access-xvf76") pod "2c215706-e61e-4f71-938f-da96ddc3e79b" (UID: "2c215706-e61e-4f71-938f-da96ddc3e79b"). InnerVolumeSpecName "kube-api-access-xvf76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:38:48.789522 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.789486    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvf76\" (UniqueName: \"kubernetes.io/projected/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-api-access-xvf76\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:38:48.789522 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.789518    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:38:48.789742 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.789532    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c215706-e61e-4f71-938f-da96ddc3e79b-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:38:48.789742 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.789543    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c215706-e61e-4f71-938f-da96ddc3e79b-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:38:48.789742 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:48.789556    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c215706-e61e-4f71-938f-da96ddc3e79b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:38:49.557777 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.557747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"
Sep 08 13:38:49.557777 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.557769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9" event={"ID":"2c215706-e61e-4f71-938f-da96ddc3e79b","Type":"ContainerDied","Data":"58c4ceba9753ea4895374212659ac730ec80b80c6abed20964d231737e84721c"}
Sep 08 13:38:49.558315 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.557857    2581 scope.go:117] "RemoveContainer" containerID="c70d9b5f05be85324430908f75dac8b594ce26d1b2d44696d7eee3feacdba822"
Sep 08 13:38:49.559351 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.559330    2581 generic.go:356] "Generic (PLEG): container finished" podID="eea0edf9-8272-4066-9693-ccbe841193f1" containerID="ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66" exitCode=0
Sep 08 13:38:49.559450 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.559367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" event={"ID":"eea0edf9-8272-4066-9693-ccbe841193f1","Type":"ContainerDied","Data":"ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66"}
Sep 08 13:38:49.566184 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.566163    2581 scope.go:117] "RemoveContainer" containerID="4e6b281a35ba77387e3306f602d3b7aaaf9e7b2ea224d3d358b03549c2e13b40"
Sep 08 13:38:49.575312 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.575281    2581 scope.go:117] "RemoveContainer" containerID="f0cf06923a435aa3ccd3e08c0034e9502ddbc6659434a90e70eb2bd1616d83a3"
Sep 08 13:38:49.576975 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.576929    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"]
Sep 08 13:38:49.581987 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:49.581967    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-73c96-predictor-577dc64498-64cq9"]
Sep 08 13:38:50.564545 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:50.564513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" event={"ID":"eea0edf9-8272-4066-9693-ccbe841193f1","Type":"ContainerStarted","Data":"c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4"}
Sep 08 13:38:50.565140 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:50.564586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" event={"ID":"eea0edf9-8272-4066-9693-ccbe841193f1","Type":"ContainerStarted","Data":"89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c"}
Sep 08 13:38:50.565140 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:50.564847    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:50.587833 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:50.587773    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podStartSLOduration=2.689348705 podStartE2EDuration="6.58775564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.898406935s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:38:50.585908257 +0000 UTC m=+1360.289694105" watchObservedRunningTime="2026-09-08 13:38:50.58775564 +0000 UTC m=+1360.291541514"
Sep 08 13:38:50.821731 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:50.821631    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" path="/var/lib/kubelet/pods/2c215706-e61e-4f71-938f-da96ddc3e79b/volumes"
Sep 08 13:38:51.569655 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:51.569622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:51.571003 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:51.570973    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:38:52.573307 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:52.573267    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:38:57.577487 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:57.577456    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:38:57.578047 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:38:57.578021    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:39:07.578499 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:39:07.578456    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:39:17.578201 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:39:17.578157    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:39:27.578567 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:39:27.578532    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:39:37.578127 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:39:37.578085    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:39:47.579010 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:39:47.578968    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:39:57.578107 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:39:57.578068    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:40:07.579457 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:07.579387    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:40:14.447622 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:14.447589    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"]
Sep 08 13:40:14.448007 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:14.447859    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" containerID="cri-o://89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c" gracePeriod=30
Sep 08 13:40:14.448007 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:14.447938    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kube-rbac-proxy" containerID="cri-o://c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4" gracePeriod=30
Sep 08 13:40:14.854893 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:14.854862    2581 generic.go:356] "Generic (PLEG): container finished" podID="eea0edf9-8272-4066-9693-ccbe841193f1" containerID="c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4" exitCode=2
Sep 08 13:40:14.855027 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:14.854932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" event={"ID":"eea0edf9-8272-4066-9693-ccbe841193f1","Type":"ContainerDied","Data":"c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4"}
Sep 08 13:40:17.573932 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:17.573893    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 08 13:40:17.578254 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:17.578224    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 08 13:40:18.393442 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.393416    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:40:18.490540 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.490449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-kube-rbac-proxy-sar-config\") pod \"eea0edf9-8272-4066-9693-ccbe841193f1\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") "
Sep 08 13:40:18.490540 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.490527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-cabundle-cert\") pod \"eea0edf9-8272-4066-9693-ccbe841193f1\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") "
Sep 08 13:40:18.490799 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.490557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd7x7\" (UniqueName: \"kubernetes.io/projected/eea0edf9-8272-4066-9693-ccbe841193f1-kube-api-access-sd7x7\") pod \"eea0edf9-8272-4066-9693-ccbe841193f1\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") "
Sep 08 13:40:18.490799 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.490606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea0edf9-8272-4066-9693-ccbe841193f1-kserve-provision-location\") pod \"eea0edf9-8272-4066-9693-ccbe841193f1\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") "
Sep 08 13:40:18.490799 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.490727    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea0edf9-8272-4066-9693-ccbe841193f1-proxy-tls\") pod \"eea0edf9-8272-4066-9693-ccbe841193f1\" (UID: \"eea0edf9-8272-4066-9693-ccbe841193f1\") "
Sep 08 13:40:18.490951 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.490887    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eea0edf9-8272-4066-9693-ccbe841193f1" (UID: "eea0edf9-8272-4066-9693-ccbe841193f1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:18.490951 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.490900    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea0edf9-8272-4066-9693-ccbe841193f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eea0edf9-8272-4066-9693-ccbe841193f1" (UID: "eea0edf9-8272-4066-9693-ccbe841193f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:18.490951 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.490941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eea0edf9-8272-4066-9693-ccbe841193f1" (UID: "eea0edf9-8272-4066-9693-ccbe841193f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:18.492750 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.492720    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea0edf9-8272-4066-9693-ccbe841193f1-kube-api-access-sd7x7" (OuterVolumeSpecName: "kube-api-access-sd7x7") pod "eea0edf9-8272-4066-9693-ccbe841193f1" (UID: "eea0edf9-8272-4066-9693-ccbe841193f1"). InnerVolumeSpecName "kube-api-access-sd7x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:40:18.493175 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.493153    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea0edf9-8272-4066-9693-ccbe841193f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eea0edf9-8272-4066-9693-ccbe841193f1" (UID: "eea0edf9-8272-4066-9693-ccbe841193f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:40:18.592158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.592131    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eea0edf9-8272-4066-9693-ccbe841193f1-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:40:18.592158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.592151    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:40:18.592158 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.592161    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eea0edf9-8272-4066-9693-ccbe841193f1-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:40:18.592531 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.592169    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd7x7\" (UniqueName: \"kubernetes.io/projected/eea0edf9-8272-4066-9693-ccbe841193f1-kube-api-access-sd7x7\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:40:18.592531 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.592177    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eea0edf9-8272-4066-9693-ccbe841193f1-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:40:18.868902 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.868870    2581 generic.go:356] "Generic (PLEG): container finished" podID="eea0edf9-8272-4066-9693-ccbe841193f1" containerID="89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c" exitCode=0
Sep 08 13:40:18.869011 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.868939    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"
Sep 08 13:40:18.869011 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.868954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" event={"ID":"eea0edf9-8272-4066-9693-ccbe841193f1","Type":"ContainerDied","Data":"89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c"}
Sep 08 13:40:18.869079 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.869018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l" event={"ID":"eea0edf9-8272-4066-9693-ccbe841193f1","Type":"ContainerDied","Data":"1775673e2b908bcf76448868a03b146f100f035282700462b06552f7477a9b75"}
Sep 08 13:40:18.869079 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.869058    2581 scope.go:117] "RemoveContainer" containerID="c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4"
Sep 08 13:40:18.876731 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.876712    2581 scope.go:117] "RemoveContainer" containerID="89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c"
Sep 08 13:40:18.884277 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.884264    2581 scope.go:117] "RemoveContainer" containerID="ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66"
Sep 08 13:40:18.891284 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.891266    2581 scope.go:117] "RemoveContainer" containerID="c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4"
Sep 08 13:40:18.891560 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:40:18.891538    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4\": container with ID starting with c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4 not found: ID does not exist" containerID="c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4"
Sep 08 13:40:18.891651 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.891565    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4"} err="failed to get container status \"c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4\": rpc error: code = NotFound desc = could not find container \"c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4\": container with ID starting with c1cec04d71c5bf0178f58bc499b44fc9e99abf381c1d5d0721dad37208e73bf4 not found: ID does not exist"
Sep 08 13:40:18.891651 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.891582    2581 scope.go:117] "RemoveContainer" containerID="89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c"
Sep 08 13:40:18.891841 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:40:18.891823    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c\": container with ID starting with 89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c not found: ID does not exist" containerID="89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c"
Sep 08 13:40:18.891887 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.891848    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c"} err="failed to get container status \"89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c\": rpc error: code = NotFound desc = could not find container \"89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c\": container with ID starting with 89fb7f4bfe2049d0d1a49b86795bfb25ff8a1d92eb69f69b59b335783fb66a3c not found: ID does not exist"
Sep 08 13:40:18.891887 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.891864    2581 scope.go:117] "RemoveContainer" containerID="ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66"
Sep 08 13:40:18.892116 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:40:18.892099    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66\": container with ID starting with ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66 not found: ID does not exist" containerID="ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66"
Sep 08 13:40:18.892159 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.892119    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66"} err="failed to get container status \"ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66\": rpc error: code = NotFound desc = could not find container \"ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66\": container with ID starting with ba96fa5704c0add1c35f63e4c5cc8371bf26f5b1c19f511c5614383631d63b66 not found: ID does not exist"
Sep 08 13:40:18.892500 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.892485    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"]
Sep 08 13:40:18.895015 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:18.894997    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-t6j5l"]
Sep 08 13:40:19.724182 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724153    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"]
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724373    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="storage-initializer"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724383    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="storage-initializer"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724391    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="storage-initializer"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724396    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="storage-initializer"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724413    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724418    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724425    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724431    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724437    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kube-rbac-proxy"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724442    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kube-rbac-proxy"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724448    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kube-rbac-proxy"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724453    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kube-rbac-proxy"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724491    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kserve-container"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724500    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kserve-container"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724504    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c215706-e61e-4f71-938f-da96ddc3e79b" containerName="kube-rbac-proxy"
Sep 08 13:40:19.724525 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.724509    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" containerName="kube-rbac-proxy"
Sep 08 13:40:19.728712 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.728697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.732279 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.732259    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 08 13:40:19.732800 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.732775    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:40:19.734359 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.734180    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 08 13:40:19.735491 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.735465    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:40:19.735723 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.735704    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:40:19.735807 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.735781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:40:19.736023 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.736004    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:40:19.740928 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.740899    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"]
Sep 08 13:40:19.801024 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.801001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfrz6\" (UniqueName: \"kubernetes.io/projected/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-api-access-hfrz6\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.801142 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.801038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.801142 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.801059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.801227 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.801144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.801227 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.801161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.804016 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.803995    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"]
Sep 08 13:40:19.807080 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.807066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:19.809248 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.809232    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 08 13:40:19.824219 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.824196    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"]
Sep 08 13:40:19.901855 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.901828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.901855 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.901856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.902021 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.901878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfrz6\" (UniqueName: \"kubernetes.io/projected/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-api-access-hfrz6\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.902021 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:40:19.901967    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 08 13:40:19.902021 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.901987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:19.902138 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4c8q\" (UniqueName: \"kubernetes.io/projected/76b39d26-028a-437e-922c-007095c453ee-kube-api-access-l4c8q\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:19.902138 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:40:19.902046    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls podName:d0c64ae8-ddf5-4c55-a10a-140a54b3c229 nodeName:}" failed. No retries permitted until 2026-09-08 13:40:20.40202422 +0000 UTC m=+1450.105810046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls") pod "isvc-canary-promote-v2-predictor-67454bd898-68xss" (UID: "d0c64ae8-ddf5-4c55-a10a-140a54b3c229") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 08 13:40:19.902138 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.902138 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:19.902322 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.902322 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b39d26-028a-437e-922c-007095c453ee-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:19.902322 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.902322 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902295    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:19.902748 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.902881 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.902865    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:19.910266 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:19.910247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfrz6\" (UniqueName: \"kubernetes.io/projected/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-api-access-hfrz6\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:20.003528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.003474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.003528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.003501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b39d26-028a-437e-922c-007095c453ee-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.003528 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.003523    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.003697 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:40:20.003616    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 08 13:40:20.003697 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:40:20.003694    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls podName:76b39d26-028a-437e-922c-007095c453ee nodeName:}" failed. No retries permitted until 2026-09-08 13:40:20.50365608 +0000 UTC m=+1450.207441907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls") pod "isvc-canary-promote-predictor-64849fbd89-8bk2k" (UID: "76b39d26-028a-437e-922c-007095c453ee") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 08 13:40:20.003800 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.003714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.003800 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.003750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4c8q\" (UniqueName: \"kubernetes.io/projected/76b39d26-028a-437e-922c-007095c453ee-kube-api-access-l4c8q\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.003800 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.003793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b39d26-028a-437e-922c-007095c453ee-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.004238 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.004218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.004300 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.004281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.016634 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.016608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4c8q\" (UniqueName: \"kubernetes.io/projected/76b39d26-028a-437e-922c-007095c453ee-kube-api-access-l4c8q\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.406519 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.406489    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:20.408864 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.408839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-68xss\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:20.507775 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.507750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.510049 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.510031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-8bk2k\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.641686 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.641626    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:20.715516 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.715488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:20.822241 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.822215    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea0edf9-8272-4066-9693-ccbe841193f1" path="/var/lib/kubelet/pods/eea0edf9-8272-4066-9693-ccbe841193f1/volumes"
Sep 08 13:40:20.936458 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.936434    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"]
Sep 08 13:40:20.939400 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:40:20.939371    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c64ae8_ddf5_4c55_a10a_140a54b3c229.slice/crio-3ec1a415846303e65bbcf77a2e2c6bca13f294f1e265d9120dc82e369193626f WatchSource:0}: Error finding container 3ec1a415846303e65bbcf77a2e2c6bca13f294f1e265d9120dc82e369193626f: Status 404 returned error can't find the container with id 3ec1a415846303e65bbcf77a2e2c6bca13f294f1e265d9120dc82e369193626f
Sep 08 13:40:20.994123 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:20.994100    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"]
Sep 08 13:40:20.997115 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:40:20.997093    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b39d26_028a_437e_922c_007095c453ee.slice/crio-4bd03e8eccff14abd5b9a5a7777d212f5196308393b6030a40f12ab0b7feb597 WatchSource:0}: Error finding container 4bd03e8eccff14abd5b9a5a7777d212f5196308393b6030a40f12ab0b7feb597: Status 404 returned error can't find the container with id 4bd03e8eccff14abd5b9a5a7777d212f5196308393b6030a40f12ab0b7feb597
Sep 08 13:40:21.881694 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:21.881640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" event={"ID":"76b39d26-028a-437e-922c-007095c453ee","Type":"ContainerStarted","Data":"451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396"}
Sep 08 13:40:21.882116 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:21.881719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" event={"ID":"76b39d26-028a-437e-922c-007095c453ee","Type":"ContainerStarted","Data":"4bd03e8eccff14abd5b9a5a7777d212f5196308393b6030a40f12ab0b7feb597"}
Sep 08 13:40:21.883204 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:21.883179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" event={"ID":"d0c64ae8-ddf5-4c55-a10a-140a54b3c229","Type":"ContainerStarted","Data":"f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94"}
Sep 08 13:40:21.883326 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:21.883222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" event={"ID":"d0c64ae8-ddf5-4c55-a10a-140a54b3c229","Type":"ContainerStarted","Data":"3ec1a415846303e65bbcf77a2e2c6bca13f294f1e265d9120dc82e369193626f"}
Sep 08 13:40:24.895066 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:24.895027    2581 generic.go:356] "Generic (PLEG): container finished" podID="76b39d26-028a-437e-922c-007095c453ee" containerID="451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396" exitCode=0
Sep 08 13:40:24.895417 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:24.895102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" event={"ID":"76b39d26-028a-437e-922c-007095c453ee","Type":"ContainerDied","Data":"451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396"}
Sep 08 13:40:24.896764 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:24.896742    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerID="f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94" exitCode=0
Sep 08 13:40:24.896854 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:24.896795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" event={"ID":"d0c64ae8-ddf5-4c55-a10a-140a54b3c229","Type":"ContainerDied","Data":"f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94"}
Sep 08 13:40:25.905215 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.905180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" event={"ID":"76b39d26-028a-437e-922c-007095c453ee","Type":"ContainerStarted","Data":"12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08"}
Sep 08 13:40:25.905692 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.905252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" event={"ID":"76b39d26-028a-437e-922c-007095c453ee","Type":"ContainerStarted","Data":"7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9"}
Sep 08 13:40:25.905692 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.905601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:25.905692 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.905658    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:25.907005 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.906973    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:40:25.907894 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.907867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" event={"ID":"d0c64ae8-ddf5-4c55-a10a-140a54b3c229","Type":"ContainerStarted","Data":"1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa"}
Sep 08 13:40:25.908020 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.907913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" event={"ID":"d0c64ae8-ddf5-4c55-a10a-140a54b3c229","Type":"ContainerStarted","Data":"c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722"}
Sep 08 13:40:25.908105 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.908088    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:25.908163 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.908142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:25.909173 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.909153    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:40:25.929073 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.929018    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podStartSLOduration=3.1744665149999998 podStartE2EDuration="6.929004735s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.75453822s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:40:25.927416753 +0000 UTC m=+1455.631202603" watchObservedRunningTime="2026-09-08 13:40:25.929004735 +0000 UTC m=+1455.632790584"
Sep 08 13:40:25.952174 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:25.952123    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podStartSLOduration=3.282806422 podStartE2EDuration="6.952108289s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.669301867s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:40:25.950280858 +0000 UTC m=+1455.654066706" watchObservedRunningTime="2026-09-08 13:40:25.952108289 +0000 UTC m=+1455.655894152"
Sep 08 13:40:26.912621 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:26.912578    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:40:26.912999 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:26.912706    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:40:31.917124 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:31.917090    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:40:31.917522 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:31.917480    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:40:31.917522 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:31.917490    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:40:31.918028 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:31.918002    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:40:41.918416 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:41.918371    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:40:41.918897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:41.918376    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:40:51.917666 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:51.917615    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:40:51.918212 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:40:51.917885    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:41:01.918117 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:01.918079    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:41:01.918497 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:01.918129    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:41:11.917561 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:11.917516    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:41:11.917961 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:11.917934    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:41:21.918177 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:21.918129    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:41:21.918548 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:21.918129    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:41:31.917854 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:31.917765    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:41:31.918342 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:31.917901    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:41:38.821727 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:38.821698    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:41:41.918796 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:41.918767    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:41:49.705887 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:49.705856    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"]
Sep 08 13:41:49.706276 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:49.706123    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container" containerID="cri-o://7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9" gracePeriod=30
Sep 08 13:41:49.706276 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:49.706171    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kube-rbac-proxy" containerID="cri-o://12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08" gracePeriod=30
Sep 08 13:41:50.211441 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:50.211405    2581 generic.go:356] "Generic (PLEG): container finished" podID="76b39d26-028a-437e-922c-007095c453ee" containerID="12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08" exitCode=2
Sep 08 13:41:50.211622 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:50.211486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" event={"ID":"76b39d26-028a-437e-922c-007095c453ee","Type":"ContainerDied","Data":"12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08"}
Sep 08 13:41:51.913638 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:51.913589    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 08 13:41:53.949377 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:53.949351    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:41:54.062666 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.062643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4c8q\" (UniqueName: \"kubernetes.io/projected/76b39d26-028a-437e-922c-007095c453ee-kube-api-access-l4c8q\") pod \"76b39d26-028a-437e-922c-007095c453ee\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") "
Sep 08 13:41:54.062815 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.062689    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-cabundle-cert\") pod \"76b39d26-028a-437e-922c-007095c453ee\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") "
Sep 08 13:41:54.062815 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.062723    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-kube-rbac-proxy-sar-config\") pod \"76b39d26-028a-437e-922c-007095c453ee\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") "
Sep 08 13:41:54.062815 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.062801    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls\") pod \"76b39d26-028a-437e-922c-007095c453ee\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") "
Sep 08 13:41:54.062951 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.062838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b39d26-028a-437e-922c-007095c453ee-kserve-provision-location\") pod \"76b39d26-028a-437e-922c-007095c453ee\" (UID: \"76b39d26-028a-437e-922c-007095c453ee\") "
Sep 08 13:41:54.063054 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.063035    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76b39d26-028a-437e-922c-007095c453ee" (UID: "76b39d26-028a-437e-922c-007095c453ee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:41:54.063229 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.063105    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76b39d26-028a-437e-922c-007095c453ee" (UID: "76b39d26-028a-437e-922c-007095c453ee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:41:54.063306 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.063243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b39d26-028a-437e-922c-007095c453ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76b39d26-028a-437e-922c-007095c453ee" (UID: "76b39d26-028a-437e-922c-007095c453ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:41:54.064888 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.064866    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76b39d26-028a-437e-922c-007095c453ee" (UID: "76b39d26-028a-437e-922c-007095c453ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:41:54.064965 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.064946    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b39d26-028a-437e-922c-007095c453ee-kube-api-access-l4c8q" (OuterVolumeSpecName: "kube-api-access-l4c8q") pod "76b39d26-028a-437e-922c-007095c453ee" (UID: "76b39d26-028a-437e-922c-007095c453ee"). InnerVolumeSpecName "kube-api-access-l4c8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:54.163236 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.163211    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76b39d26-028a-437e-922c-007095c453ee-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:41:54.163236 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.163231    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76b39d26-028a-437e-922c-007095c453ee-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:41:54.163367 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.163241    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4c8q\" (UniqueName: \"kubernetes.io/projected/76b39d26-028a-437e-922c-007095c453ee-kube-api-access-l4c8q\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:41:54.163367 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.163249    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:41:54.163367 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.163258    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76b39d26-028a-437e-922c-007095c453ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:41:54.227097 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.227072    2581 generic.go:356] "Generic (PLEG): container finished" podID="76b39d26-028a-437e-922c-007095c453ee" containerID="7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9" exitCode=0
Sep 08 13:41:54.227211 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.227129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" event={"ID":"76b39d26-028a-437e-922c-007095c453ee","Type":"ContainerDied","Data":"7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9"}
Sep 08 13:41:54.227211 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.227147    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"
Sep 08 13:41:54.227211 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.227174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k" event={"ID":"76b39d26-028a-437e-922c-007095c453ee","Type":"ContainerDied","Data":"4bd03e8eccff14abd5b9a5a7777d212f5196308393b6030a40f12ab0b7feb597"}
Sep 08 13:41:54.227211 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.227200    2581 scope.go:117] "RemoveContainer" containerID="12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08"
Sep 08 13:41:54.235557 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.235540    2581 scope.go:117] "RemoveContainer" containerID="7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9"
Sep 08 13:41:54.242524 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.242506    2581 scope.go:117] "RemoveContainer" containerID="451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396"
Sep 08 13:41:54.249031 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.249010    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"]
Sep 08 13:41:54.250208 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.250192    2581 scope.go:117] "RemoveContainer" containerID="12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08"
Sep 08 13:41:54.250492 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:41:54.250474    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08\": container with ID starting with 12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08 not found: ID does not exist" containerID="12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08"
Sep 08 13:41:54.250596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.250506    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08"} err="failed to get container status \"12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08\": rpc error: code = NotFound desc = could not find container \"12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08\": container with ID starting with 12806d1ff495c9cb157c5a37f8a4c985986211704d72d71ff84f0cbdc994bd08 not found: ID does not exist"
Sep 08 13:41:54.250596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.250526    2581 scope.go:117] "RemoveContainer" containerID="7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9"
Sep 08 13:41:54.251234 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:41:54.251194    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9\": container with ID starting with 7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9 not found: ID does not exist" containerID="7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9"
Sep 08 13:41:54.251303 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.251227    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9"} err="failed to get container status \"7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9\": rpc error: code = NotFound desc = could not find container \"7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9\": container with ID starting with 7223004c2e4d7336fb61e07826d5dc0546d8be5b5b8bce748bd85be79be5d3e9 not found: ID does not exist"
Sep 08 13:41:54.251303 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.251247    2581 scope.go:117] "RemoveContainer" containerID="451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396"
Sep 08 13:41:54.251544 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:41:54.251525    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396\": container with ID starting with 451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396 not found: ID does not exist" containerID="451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396"
Sep 08 13:41:54.251593 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.251548    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396"} err="failed to get container status \"451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396\": rpc error: code = NotFound desc = could not find container \"451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396\": container with ID starting with 451267565e3d8f71dac4c06bd98f358e0b2e65e0b6aaba05bd3771e3dea34396 not found: ID does not exist"
Sep 08 13:41:54.252105 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.252088    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-8bk2k"]
Sep 08 13:41:54.822486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:41:54.822453    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b39d26-028a-437e-922c-007095c453ee" path="/var/lib/kubelet/pods/76b39d26-028a-437e-922c-007095c453ee/volumes"
Sep 08 13:43:20.157438 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:20.157359    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"]
Sep 08 13:43:20.158014 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:20.157588    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" containerID="cri-o://c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722" gracePeriod=30
Sep 08 13:43:20.158014 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:20.157635    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kube-rbac-proxy" containerID="cri-o://1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa" gracePeriod=30
Sep 08 13:43:20.528293 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:20.528208    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerID="1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa" exitCode=2
Sep 08 13:43:20.528293 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:20.528276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" event={"ID":"d0c64ae8-ddf5-4c55-a10a-140a54b3c229","Type":"ContainerDied","Data":"1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa"}
Sep 08 13:43:21.913784 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:21.913736    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 08 13:43:21.917846 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:21.917821    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Sep 08 13:43:24.395223 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.395192    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:43:24.544137 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.544105    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerID="c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722" exitCode=0
Sep 08 13:43:24.544310 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.544174    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"
Sep 08 13:43:24.544310 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.544195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" event={"ID":"d0c64ae8-ddf5-4c55-a10a-140a54b3c229","Type":"ContainerDied","Data":"c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722"}
Sep 08 13:43:24.544310 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.544286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss" event={"ID":"d0c64ae8-ddf5-4c55-a10a-140a54b3c229","Type":"ContainerDied","Data":"3ec1a415846303e65bbcf77a2e2c6bca13f294f1e265d9120dc82e369193626f"}
Sep 08 13:43:24.544468 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.544315    2581 scope.go:117] "RemoveContainer" containerID="1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa"
Sep 08 13:43:24.552044 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.552021    2581 scope.go:117] "RemoveContainer" containerID="c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722"
Sep 08 13:43:24.558768 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.558746    2581 scope.go:117] "RemoveContainer" containerID="f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94"
Sep 08 13:43:24.565488 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.565469    2581 scope.go:117] "RemoveContainer" containerID="1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa"
Sep 08 13:43:24.565754 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:43:24.565734    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa\": container with ID starting with 1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa not found: ID does not exist" containerID="1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa"
Sep 08 13:43:24.565829 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.565767    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa"} err="failed to get container status \"1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa\": rpc error: code = NotFound desc = could not find container \"1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa\": container with ID starting with 1c70a9f3bbb9f280913cd8f9211804996daaecb2f3cc438ab9b32c0a7c3dc1aa not found: ID does not exist"
Sep 08 13:43:24.565829 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.565791    2581 scope.go:117] "RemoveContainer" containerID="c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722"
Sep 08 13:43:24.566051 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:43:24.566035    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722\": container with ID starting with c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722 not found: ID does not exist" containerID="c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722"
Sep 08 13:43:24.566099 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.566058    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722"} err="failed to get container status \"c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722\": rpc error: code = NotFound desc = could not find container \"c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722\": container with ID starting with c89485bd3037c75b512f287e774dcaa8635018beb445400868bc0a2df7d27722 not found: ID does not exist"
Sep 08 13:43:24.566099 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.566075    2581 scope.go:117] "RemoveContainer" containerID="f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94"
Sep 08 13:43:24.566321 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:43:24.566303    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94\": container with ID starting with f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94 not found: ID does not exist" containerID="f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94"
Sep 08 13:43:24.566366 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.566327    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94"} err="failed to get container status \"f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94\": rpc error: code = NotFound desc = could not find container \"f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94\": container with ID starting with f790dc59445d5eebef78f95e4e6b7f4a82f35833f22dabaa2a33a679e3390e94 not found: ID does not exist"
Sep 08 13:43:24.579569 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.579550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfrz6\" (UniqueName: \"kubernetes.io/projected/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-api-access-hfrz6\") pod \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") "
Sep 08 13:43:24.579660 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.579600    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kserve-provision-location\") pod \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") "
Sep 08 13:43:24.579660 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.579637    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls\") pod \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") "
Sep 08 13:43:24.579789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.579696    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-cabundle-cert\") pod \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") "
Sep 08 13:43:24.579789 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.579754    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-rbac-proxy-sar-config\") pod \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\" (UID: \"d0c64ae8-ddf5-4c55-a10a-140a54b3c229\") "
Sep 08 13:43:24.579946 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.579918    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0c64ae8-ddf5-4c55-a10a-140a54b3c229" (UID: "d0c64ae8-ddf5-4c55-a10a-140a54b3c229"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:43:24.580067 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.580044    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d0c64ae8-ddf5-4c55-a10a-140a54b3c229" (UID: "d0c64ae8-ddf5-4c55-a10a-140a54b3c229"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:43:24.580123 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.580106    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0c64ae8-ddf5-4c55-a10a-140a54b3c229" (UID: "d0c64ae8-ddf5-4c55-a10a-140a54b3c229"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:43:24.581625 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.581609    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-api-access-hfrz6" (OuterVolumeSpecName: "kube-api-access-hfrz6") pod "d0c64ae8-ddf5-4c55-a10a-140a54b3c229" (UID: "d0c64ae8-ddf5-4c55-a10a-140a54b3c229"). InnerVolumeSpecName "kube-api-access-hfrz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:43:24.581749 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.581730    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0c64ae8-ddf5-4c55-a10a-140a54b3c229" (UID: "d0c64ae8-ddf5-4c55-a10a-140a54b3c229"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:43:24.680810 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.680780    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:43:24.680810 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.680805    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:43:24.680810 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.680815    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfrz6\" (UniqueName: \"kubernetes.io/projected/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kube-api-access-hfrz6\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:43:24.680993 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.680823    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:43:24.680993 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.680832    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0c64ae8-ddf5-4c55-a10a-140a54b3c229-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:43:24.868484 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.868452    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"]
Sep 08 13:43:24.871989 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:24.871967    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-68xss"]
Sep 08 13:43:26.821649 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:26.821613    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" path="/var/lib/kubelet/pods/d0c64ae8-ddf5-4c55-a10a-140a54b3c229/volumes"
Sep 08 13:43:44.266178 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266136    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"]
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266438    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266453    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266466    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kube-rbac-proxy"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266471    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kube-rbac-proxy"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266478    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266484    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266490    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kube-rbac-proxy"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266495    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kube-rbac-proxy"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266503    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="storage-initializer"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266508    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="storage-initializer"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266519    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="storage-initializer"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266524    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="storage-initializer"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266558    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kserve-container"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266565    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c64ae8-ddf5-4c55-a10a-140a54b3c229" containerName="kube-rbac-proxy"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266570    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kube-rbac-proxy"
Sep 08 13:43:44.266596 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.266577    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b39d26-028a-437e-922c-007095c453ee" containerName="kserve-container"
Sep 08 13:43:44.269506 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.269491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.272008 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.271980    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:43:44.272148 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.272019    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:43:44.272148 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.272072    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\""
Sep 08 13:43:44.272148 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.272099    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:43:44.272148 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.272075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-48hwc\""
Sep 08 13:43:44.272365 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.272351    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\""
Sep 08 13:43:44.272804 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.272788    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:43:44.291572 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.291544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"]
Sep 08 13:43:44.304798 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.304775    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff594022-599f-415f-a201-31b4bba772aa-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.304910 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.304802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdsw\" (UniqueName: \"kubernetes.io/projected/ff594022-599f-415f-a201-31b4bba772aa-kube-api-access-mcdsw\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.304910 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.304841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff594022-599f-415f-a201-31b4bba772aa-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.305024 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.304900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.305024 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.304947    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.405517 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.405488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff594022-599f-415f-a201-31b4bba772aa-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.405696 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.405519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.405696 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.405555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.405696 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.405603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff594022-599f-415f-a201-31b4bba772aa-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.405696 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.405630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdsw\" (UniqueName: \"kubernetes.io/projected/ff594022-599f-415f-a201-31b4bba772aa-kube-api-access-mcdsw\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.406084 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.406052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff594022-599f-415f-a201-31b4bba772aa-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.406258 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.406237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.406327 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.406275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.408048 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.408019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff594022-599f-415f-a201-31b4bba772aa-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.414731 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.414709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdsw\" (UniqueName: \"kubernetes.io/projected/ff594022-599f-415f-a201-31b4bba772aa-kube-api-access-mcdsw\") pod \"isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.578944 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.578908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:44.873281 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:44.873233    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"]
Sep 08 13:43:44.875435 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:43:44.875407    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff594022_599f_415f_a201_31b4bba772aa.slice/crio-096d7df6aa1a60ee329d3f4e0ebf936295950337fc1ccfdbcdab594b84002451 WatchSource:0}: Error finding container 096d7df6aa1a60ee329d3f4e0ebf936295950337fc1ccfdbcdab594b84002451: Status 404 returned error can't find the container with id 096d7df6aa1a60ee329d3f4e0ebf936295950337fc1ccfdbcdab594b84002451
Sep 08 13:43:45.618072 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:45.618039    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerStarted","Data":"1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f"}
Sep 08 13:43:45.618459 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:45.618101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerStarted","Data":"096d7df6aa1a60ee329d3f4e0ebf936295950337fc1ccfdbcdab594b84002451"}
Sep 08 13:43:48.630357 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:48.630323    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff594022-599f-415f-a201-31b4bba772aa" containerID="1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f" exitCode=0
Sep 08 13:43:48.630652 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:48.630400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerDied","Data":"1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f"}
Sep 08 13:43:48.749142 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:48.749123    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:43:49.636949 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:49.636917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerStarted","Data":"d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5"}
Sep 08 13:43:49.637322 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:49.636976    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerStarted","Data":"c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9"}
Sep 08 13:43:53.653265 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:53.653218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerStarted","Data":"484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84"}
Sep 08 13:43:53.653714 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:53.653486    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:53.653714 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:53.653582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:53.655001 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:53.654970    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:43:53.658410 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:53.658389    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:53.679116 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:53.679074    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podStartSLOduration=2.170419808 podStartE2EDuration="9.679060278s" totalImagesPullingTime="3.867973679s" totalInitContainerRuntime="3.640666791s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:43:53.677390615 +0000 UTC m=+1663.381176503" watchObservedRunningTime="2026-09-08 13:43:53.679060278 +0000 UTC m=+1663.382846127"
Sep 08 13:43:54.657147 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:54.657109    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:43:54.657618 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:54.657341    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:43:54.658221 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:54.658195    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:43:55.661420 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:55.661377    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:43:55.661845 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:43:55.661738    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:44:05.662004 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:05.661956    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:44:05.662534 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:05.662508    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:44:15.661787 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:15.661737    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:44:15.662266 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:15.662180    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:44:25.661614 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:25.661555    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:44:25.662142 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:25.661977    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:44:35.662039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:35.661947    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:44:35.662431 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:35.662406    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:44:45.661621 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:45.661578    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:44:45.662218 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:45.662194    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:44:55.661916 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:55.661861    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:44:55.662492 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:55.662378    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:44:58.822428 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:58.822396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:44:58.822879 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:44:58.822497    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:45:09.533388 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:09.533353    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"]
Sep 08 13:45:09.533801 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:09.533598    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" containerID="cri-o://c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9" gracePeriod=30
Sep 08 13:45:09.533801 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:09.533630    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" containerID="cri-o://484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84" gracePeriod=30
Sep 08 13:45:09.533801 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:09.533683    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy" containerID="cri-o://d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5" gracePeriod=30
Sep 08 13:45:09.915045 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:09.915013    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff594022-599f-415f-a201-31b4bba772aa" containerID="d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5" exitCode=2
Sep 08 13:45:09.915190 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:09.915071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerDied","Data":"d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5"}
Sep 08 13:45:13.654792 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:13.654754    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 08 13:45:13.931655 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:13.931628    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff594022-599f-415f-a201-31b4bba772aa" containerID="c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9" exitCode=0
Sep 08 13:45:13.931798 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:13.931701    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerDied","Data":"c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9"}
Sep 08 13:45:18.654350 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:18.654306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 08 13:45:18.818605 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:18.818560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:45:18.818977 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:18.818952    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:45:23.654367 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:23.654322    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 08 13:45:23.654776 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:23.654454    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:45:26.198929 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:26.198902    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:26.233301 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:26.233274    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:26.252588 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:26.252569    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:26.265092 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:26.265065    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:27.126754 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:27.126725    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:27.142893 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:27.142869    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:27.167034 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:27.167011    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:27.181463 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:27.181441    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:28.023273 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.023244    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:28.036156 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.036131    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:28.047249 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.047228    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:28.058796 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.058777    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:28.654039 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.653999    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 08 13:45:28.818301 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.818258    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:45:28.818595 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.818554    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:45:28.883752 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.883729    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:28.899256 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.899228    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:28.910295 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.910213    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:28.921585 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:28.921552    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:29.764123 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:29.764085    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:29.779694 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:29.779640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:29.791434 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:29.791412    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:29.802246 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:29.802205    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:30.654165 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:30.654135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:30.667011 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:30.666975    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:30.678120 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:30.678095    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:30.692072 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:30.692057    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:31.514279 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:31.514243    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:31.530770 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:31.530742    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:31.542173 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:31.542152    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:31.553486 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:31.553466    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:32.374827 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:32.374786    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:32.387251 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:32.387219    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:32.399278 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:32.399253    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:32.408772 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:32.408747    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:33.235558 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:33.235531    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:33.249844 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:33.249823    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:33.262378 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:33.262355    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:33.279495 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:33.279477    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:33.653740 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:33.653702    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 08 13:45:34.141391 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:34.141345    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:34.168335 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:34.168312    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:34.196233 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:34.196209    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:34.224704 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:34.224663    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:35.073557 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:35.073524    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:35.087648 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:35.087626    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:35.102315 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:35.102295    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:35.114897 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:35.114873    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:35.984435 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:35.984407    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:36.001738 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:36.001716    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:36.022786 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:36.022767    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:36.041225 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:36.041193    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:36.957303 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:36.957277    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:36.972279 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:36.972236    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:36.991898 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:36.991878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:37.004496 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:37.004478    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:37.969680 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:37.969648    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kserve-container/0.log"
Sep 08 13:45:37.983854 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:37.983830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/kube-rbac-proxy/0.log"
Sep 08 13:45:37.995031 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:37.994999    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/agent/0.log"
Sep 08 13:45:38.006593 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:38.006573    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2_ff594022-599f-415f-a201-31b4bba772aa/storage-initializer/0.log"
Sep 08 13:45:38.654613 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:38.654568    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 08 13:45:38.818635 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:38.818595    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:45:38.819088 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:38.819039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 13:45:38.821095 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:38.821073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:45:38.821187 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:38.821176    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:45:39.675256 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.675230    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:45:39.793391 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.793366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-cabundle-cert\") pod \"ff594022-599f-415f-a201-31b4bba772aa\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") "
Sep 08 13:45:39.793556 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.793404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff594022-599f-415f-a201-31b4bba772aa-kserve-provision-location\") pod \"ff594022-599f-415f-a201-31b4bba772aa\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") "
Sep 08 13:45:39.793556 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.793437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-kube-rbac-proxy-sar-config\") pod \"ff594022-599f-415f-a201-31b4bba772aa\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") "
Sep 08 13:45:39.793556 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.793456    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcdsw\" (UniqueName: \"kubernetes.io/projected/ff594022-599f-415f-a201-31b4bba772aa-kube-api-access-mcdsw\") pod \"ff594022-599f-415f-a201-31b4bba772aa\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") "
Sep 08 13:45:39.793556 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.793476    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff594022-599f-415f-a201-31b4bba772aa-proxy-tls\") pod \"ff594022-599f-415f-a201-31b4bba772aa\" (UID: \"ff594022-599f-415f-a201-31b4bba772aa\") "
Sep 08 13:45:39.793753 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.793704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff594022-599f-415f-a201-31b4bba772aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff594022-599f-415f-a201-31b4bba772aa" (UID: "ff594022-599f-415f-a201-31b4bba772aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:39.793800 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.793760    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff594022-599f-415f-a201-31b4bba772aa" (UID: "ff594022-599f-415f-a201-31b4bba772aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:39.793858 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.793839    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff594022-599f-415f-a201-31b4bba772aa" (UID: "ff594022-599f-415f-a201-31b4bba772aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:39.795741 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.795709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff594022-599f-415f-a201-31b4bba772aa-kube-api-access-mcdsw" (OuterVolumeSpecName: "kube-api-access-mcdsw") pod "ff594022-599f-415f-a201-31b4bba772aa" (UID: "ff594022-599f-415f-a201-31b4bba772aa"). InnerVolumeSpecName "kube-api-access-mcdsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:39.795865 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.795768    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff594022-599f-415f-a201-31b4bba772aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff594022-599f-415f-a201-31b4bba772aa" (UID: "ff594022-599f-415f-a201-31b4bba772aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:45:39.894478 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.894442    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-cabundle-cert\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:45:39.894478 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.894476    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff594022-599f-415f-a201-31b4bba772aa-kserve-provision-location\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:45:39.894737 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.894487    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff594022-599f-415f-a201-31b4bba772aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:45:39.894737 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.894498    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdsw\" (UniqueName: \"kubernetes.io/projected/ff594022-599f-415f-a201-31b4bba772aa-kube-api-access-mcdsw\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:45:39.894737 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:39.894508    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff594022-599f-415f-a201-31b4bba772aa-proxy-tls\") on node \"ip-10-0-136-211.ec2.internal\" DevicePath \"\""
Sep 08 13:45:40.026835 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.026803    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff594022-599f-415f-a201-31b4bba772aa" containerID="484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84" exitCode=0
Sep 08 13:45:40.026979 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.026839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerDied","Data":"484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84"}
Sep 08 13:45:40.026979 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.026883    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"
Sep 08 13:45:40.026979 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.026923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2" event={"ID":"ff594022-599f-415f-a201-31b4bba772aa","Type":"ContainerDied","Data":"096d7df6aa1a60ee329d3f4e0ebf936295950337fc1ccfdbcdab594b84002451"}
Sep 08 13:45:40.026979 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.026974    2581 scope.go:117] "RemoveContainer" containerID="484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84"
Sep 08 13:45:40.037726 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.037581    2581 scope.go:117] "RemoveContainer" containerID="d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5"
Sep 08 13:45:40.044914 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.044899    2581 scope.go:117] "RemoveContainer" containerID="c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9"
Sep 08 13:45:40.051613 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.051597    2581 scope.go:117] "RemoveContainer" containerID="1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f"
Sep 08 13:45:40.055389 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.055370    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"]
Sep 08 13:45:40.058709 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.058693    2581 scope.go:117] "RemoveContainer" containerID="484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84"
Sep 08 13:45:40.058976 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:45:40.058947    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84\": container with ID starting with 484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84 not found: ID does not exist" containerID="484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84"
Sep 08 13:45:40.059071 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.058988    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84"} err="failed to get container status \"484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84\": rpc error: code = NotFound desc = could not find container \"484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84\": container with ID starting with 484d3bcde7585f6b2e904ea1c2b1a3716aa2450101748be4268ce9945a600d84 not found: ID does not exist"
Sep 08 13:45:40.059071 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.059012    2581 scope.go:117] "RemoveContainer" containerID="d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5"
Sep 08 13:45:40.059351 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:45:40.059327    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5\": container with ID starting with d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5 not found: ID does not exist" containerID="d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5"
Sep 08 13:45:40.059435 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.059361    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5"} err="failed to get container status \"d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5\": rpc error: code = NotFound desc = could not find container \"d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5\": container with ID starting with d698b7a44b60a865e0e5b49c188d2736ff82f87cbb52557b149c9735c95190e5 not found: ID does not exist"
Sep 08 13:45:40.059435 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.059382    2581 scope.go:117] "RemoveContainer" containerID="c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9"
Sep 08 13:45:40.059740 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:45:40.059715    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9\": container with ID starting with c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9 not found: ID does not exist" containerID="c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9"
Sep 08 13:45:40.059832 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.059746    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9"} err="failed to get container status \"c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9\": rpc error: code = NotFound desc = could not find container \"c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9\": container with ID starting with c5badd9d0414851111726b4c5b50d05c73951bf25e3da9b4b7032b5c40bc26d9 not found: ID does not exist"
Sep 08 13:45:40.059832 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.059766    2581 scope.go:117] "RemoveContainer" containerID="1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f"
Sep 08 13:45:40.060297 ip-10-0-136-211 kubenswrapper[2581]: E0908 13:45:40.060276    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f\": container with ID starting with 1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f not found: ID does not exist" containerID="1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f"
Sep 08 13:45:40.060353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.060306    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f"} err="failed to get container status \"1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f\": rpc error: code = NotFound desc = could not find container \"1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f\": container with ID starting with 1a37811ab53dd4b7a6904c1f47de5fe1de2425f1e6d0903d650bd80aaa9f191f not found: ID does not exist"
Sep 08 13:45:40.061011 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.060994    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-5fc9c469d5-9cjl2"]
Sep 08 13:45:40.821767 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:40.821737    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff594022-599f-415f-a201-31b4bba772aa" path="/var/lib/kubelet/pods/ff594022-599f-415f-a201-31b4bba772aa/volumes"
Sep 08 13:45:44.420806 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:44.420779    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lfltj_02436e3b-bde9-4934-98a4-91de9fec7159/global-pull-secret-syncer/0.log"
Sep 08 13:45:44.564992 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:44.564958    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wnrbm_c7e1ca70-26c8-46d3-ad76-7bea5f11a16c/konnectivity-agent/0.log"
Sep 08 13:45:44.607454 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:44.607432    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-211.ec2.internal_4084ecab986bde65b84b8c14d785fe7d/haproxy/0.log"
Sep 08 13:45:49.337399 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:49.337363    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dd9rn_f7dd71c4-df9d-44a4-89f4-030403050ffc/node-exporter/0.log"
Sep 08 13:45:49.360881 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:49.360854    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dd9rn_f7dd71c4-df9d-44a4-89f4-030403050ffc/kube-rbac-proxy/0.log"
Sep 08 13:45:49.391919 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:49.391896    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dd9rn_f7dd71c4-df9d-44a4-89f4-030403050ffc/init-textfile/0.log"
Sep 08 13:45:53.017618 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017580    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"]
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017814    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="storage-initializer"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017826    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="storage-initializer"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017838    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017844    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017853    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017858    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017867    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017873    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017913    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kserve-container"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017920    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="kube-rbac-proxy"
Sep 08 13:45:53.017984 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.017927    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff594022-599f-415f-a201-31b4bba772aa" containerName="agent"
Sep 08 13:45:53.020610 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.020593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.023306 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.023276    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r7425\"/\"kube-root-ca.crt\""
Sep 08 13:45:53.023859 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.023841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r7425\"/\"default-dockercfg-zfwhh\""
Sep 08 13:45:53.023958 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.023875    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r7425\"/\"openshift-service-ca.crt\""
Sep 08 13:45:53.041904 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.041879    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"]
Sep 08 13:45:53.081229 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.081200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-lib-modules\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.081353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.081239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pv6n\" (UniqueName: \"kubernetes.io/projected/11d607b0-a6a6-43df-a533-012ecb819368-kube-api-access-5pv6n\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.081353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.081264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-proc\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.081353 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.081329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-sys\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.081460 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.081352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-podres\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.181902 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.181877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-proc\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.182020 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.181909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-sys\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.182020 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.181932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-podres\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.182020 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.181961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-lib-modules\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.182020 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.181986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pv6n\" (UniqueName: \"kubernetes.io/projected/11d607b0-a6a6-43df-a533-012ecb819368-kube-api-access-5pv6n\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.182214 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.182028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-sys\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.182214 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.182033    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-proc\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.182214 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.182072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-podres\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.182214 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.182101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11d607b0-a6a6-43df-a533-012ecb819368-lib-modules\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.191359 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.191336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pv6n\" (UniqueName: \"kubernetes.io/projected/11d607b0-a6a6-43df-a533-012ecb819368-kube-api-access-5pv6n\") pod \"perf-node-gather-daemonset-z27r7\" (UID: \"11d607b0-a6a6-43df-a533-012ecb819368\") " pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.330262 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.330240    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:53.607092 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:53.607020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"]
Sep 08 13:45:53.611284 ip-10-0-136-211 kubenswrapper[2581]: W0908 13:45:53.611257    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod11d607b0_a6a6_43df_a533_012ecb819368.slice/crio-16b92885c59f0bf5c6e4d5eb3f5ac1c3268a5c2bc8191064519dc53efce76c4f WatchSource:0}: Error finding container 16b92885c59f0bf5c6e4d5eb3f5ac1c3268a5c2bc8191064519dc53efce76c4f: Status 404 returned error can't find the container with id 16b92885c59f0bf5c6e4d5eb3f5ac1c3268a5c2bc8191064519dc53efce76c4f
Sep 08 13:45:54.077002 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:54.076969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7" event={"ID":"11d607b0-a6a6-43df-a533-012ecb819368","Type":"ContainerStarted","Data":"5661ad7b64a10827d647c707feef6af1234923ea760e8cad51ee0c79304ab6f9"}
Sep 08 13:45:54.077373 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:54.077019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7" event={"ID":"11d607b0-a6a6-43df-a533-012ecb819368","Type":"ContainerStarted","Data":"16b92885c59f0bf5c6e4d5eb3f5ac1c3268a5c2bc8191064519dc53efce76c4f"}
Sep 08 13:45:54.077373 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:54.077073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:45:54.102516 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:54.102471    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7" podStartSLOduration=2.1024576 podStartE2EDuration="2.1024576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:45:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:45:54.101807207 +0000 UTC m=+1783.805593070" watchObservedRunningTime="2026-09-08 13:45:54.1024576 +0000 UTC m=+1783.806243448"
Sep 08 13:45:54.456539 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:54.456468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ntxdb_c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810/dns/0.log"
Sep 08 13:45:54.480636 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:54.480616    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ntxdb_c1c1d41a-1e5d-4967-b6ee-7e05f2ce0810/kube-rbac-proxy/0.log"
Sep 08 13:45:54.581269 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:54.581251    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bxllr_2f80b4d3-387c-4580-9899-0be849014b1e/dns-node-resolver/0.log"
Sep 08 13:45:55.169704 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:55.169661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lnvw8_d3fd4abc-ff7d-4512-a9f2-8103510ca5c6/node-ca/0.log"
Sep 08 13:45:56.395282 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:56.395253    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qx8ch_a1e7cc14-c9c0-4576-931e-53bc31b9326b/serve-healthcheck-canary/0.log"
Sep 08 13:45:57.009549 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:57.009519    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zn476_765018f8-64fc-4b09-9a08-a9e65fb049f0/kube-rbac-proxy/0.log"
Sep 08 13:45:57.033982 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:57.033958    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zn476_765018f8-64fc-4b09-9a08-a9e65fb049f0/exporter/0.log"
Sep 08 13:45:57.056598 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:57.056575    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zn476_765018f8-64fc-4b09-9a08-a9e65fb049f0/extractor/0.log"
Sep 08 13:45:59.524558 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:45:59.524523    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6f9bbc7b7c-bv94t_60b6a7a2-eb8d-42ed-9863-493653caf289/manager/0.log"
Sep 08 13:46:00.090527 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:00.090499    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r7425/perf-node-gather-daemonset-z27r7"
Sep 08 13:46:06.317641 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.317609    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgl7l_93bed6de-bafb-4a04-aa8b-2a4f94ff1497/kube-multus-additional-cni-plugins/0.log"
Sep 08 13:46:06.348795 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.348772    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgl7l_93bed6de-bafb-4a04-aa8b-2a4f94ff1497/egress-router-binary-copy/0.log"
Sep 08 13:46:06.377232 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.377196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgl7l_93bed6de-bafb-4a04-aa8b-2a4f94ff1497/cni-plugins/0.log"
Sep 08 13:46:06.412337 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.412315    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgl7l_93bed6de-bafb-4a04-aa8b-2a4f94ff1497/bond-cni-plugin/0.log"
Sep 08 13:46:06.444540 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.444515    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgl7l_93bed6de-bafb-4a04-aa8b-2a4f94ff1497/routeoverride-cni/0.log"
Sep 08 13:46:06.473840 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.473822    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgl7l_93bed6de-bafb-4a04-aa8b-2a4f94ff1497/whereabouts-cni-bincopy/0.log"
Sep 08 13:46:06.506344 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.506322    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sgl7l_93bed6de-bafb-4a04-aa8b-2a4f94ff1497/whereabouts-cni/0.log"
Sep 08 13:46:06.554688 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.554644    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g9wzq_b804ad43-3ef5-4847-914e-456974ad6d9b/kube-multus/0.log"
Sep 08 13:46:06.723250 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.723176    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mpqn2_1a5ec803-ab0d-4166-bf48-77a72a26b284/network-metrics-daemon/0.log"
Sep 08 13:46:06.752301 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:06.752283    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mpqn2_1a5ec803-ab0d-4166-bf48-77a72a26b284/kube-rbac-proxy/0.log"
Sep 08 13:46:07.533106 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:07.533079    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2n4x_0c2245f7-7abe-45bd-8ab1-d500057643ff/ovn-controller/0.log"
Sep 08 13:46:07.549870 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:07.549839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2n4x_0c2245f7-7abe-45bd-8ab1-d500057643ff/ovn-acl-logging/0.log"
Sep 08 13:46:07.581177 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:07.581155    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2n4x_0c2245f7-7abe-45bd-8ab1-d500057643ff/kube-rbac-proxy-node/0.log"
Sep 08 13:46:07.607627 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:07.607602    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2n4x_0c2245f7-7abe-45bd-8ab1-d500057643ff/kube-rbac-proxy-ovn-metrics/0.log"
Sep 08 13:46:07.626364 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:07.626341    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2n4x_0c2245f7-7abe-45bd-8ab1-d500057643ff/northd/0.log"
Sep 08 13:46:07.648400 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:07.648380    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2n4x_0c2245f7-7abe-45bd-8ab1-d500057643ff/nbdb/0.log"
Sep 08 13:46:07.672535 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:07.672501    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2n4x_0c2245f7-7abe-45bd-8ab1-d500057643ff/sbdb/0.log"
Sep 08 13:46:07.726003 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:07.725981    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-c2n4x_0c2245f7-7abe-45bd-8ab1-d500057643ff/ovnkube-controller/0.log"
Sep 08 13:46:09.299220 ip-10-0-136-211 kubenswrapper[2581]: I0908 13:46:09.299194    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-f7wdv_cd92145f-03d4-44bc-ab70-24feb2c337f4/network-check-target-container/0.log"